filebeat导入Hadoop日志到Elasticsearch中
filebeat导入Hadoop日志到Elasticsearch中
最近工作需要，用filebeat将Hadoop日志导入到Elasticsearch中，在kibana中展示，记录下。
版本分别是elasticsearch:6.5.1、kibana:6.5.1、filebeat:6.4，版本要对应，开始我用的filebeat版本是7.0，导致在kibana中展示时有问题，后来用6.4版本就没有这个问题了。
首先要启动elasticearch和kibana，我是在Docker中运行这俩个的，具体可以看我的elasticsearch、kibana博客。浏览器打开localhost:5601可以看到kibana的界面。
去filebeat官网下载6.4版本的压缩包，之后解压并进入目录，之后修改filebeat.yml:
List-1
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log
- /opt/software/tool/hadoop/hadoop/logs/*.log
#- c:\programdata\elasticsearch\logs\*修改filebeat的input：
- enabled的值改为true
- paths的值添加hadoop的日志目录
List-2
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "localhost:5601"如上List-2中所示，修改Kibana模块，将'host: "localhost:5601"'之前的注释去掉。
List-3
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"如List-3所示，修改Outputs模块，将Elasticsearch output的hosts: "localhost:9200"的值改为我们自己elasticsearch的IP与端口。
之后如List-4所示，没有报错就说明成功了。
List-4
./filebeat setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
#启动filebeat
./filebeat -e -c filebeat.yml来看kibana的界面，如下图1中的Discover可以看到日志，图2中的logs可以看到随着hadoop的日志文件内容被修改，kibana中的日志会相应的滚动显示，图1和图2中的搜索框中可以输入值进行搜索。
图1
图2
(adsbygoogle = window.adsbygoogle || []).push({});