【ES三周年】在Docker环境下部署EFK日志收集系统

原创

江湖有缘

修改于 2023-07-27 17:38:41

2.4K0

修改于 2023-07-27 17:38:41

文章被收录于专栏：Linux成长之路Linux成长之路

一、EFK日志收集系统介绍

1.EFK简介

EFK：分别表示：Elasticsearch , Filebeat, Kibana , 其中ELasticsearch负责日志保存和搜索，Filebeat负责收集日志，Kibana 负责界面，三者配合起来，形成一个非常完美的解决方案。

2.Elasticsearch简介

Elasticsearch是一个基于Apache Lucene的开源搜索和数据分析引擎引擎，Elasticsearch使用Java进行开发，并使用Lucene作为其核心实现所有索引和搜索的功能。

3.Elasticsearch的特点

Elasticsearch是一个实时的，分布式的，可扩展的搜索引擎。
Elasticsearch允许进行全文本和结构化搜索以及对日志进行分析。
Elasticsearch 是一个搜索引擎，负责存储日志并提供查询接口。
Elasticsearch通常用于索引和搜索大量日志数据，也可以用于搜索许多不同种类的文档。

4.、Filebeat与Kibana介绍

Filebeat是用于转发和集中日志数据的轻量级传送工具。Filebeat监视您指定的日志文件或位置，收集日志事件，并将它们转发到Elasticsearch或 Logstash进行索引。
Kibana是一个开源的分析与可视化平台，被设计用于和Elasticsearch一起使用的。通过kibana可以搜索、查看和交互存放在Elasticsearch中的数据，利用各种不同的图表、表格和地图等，Kibana能够对数据进行分析与可视化。

5.EFK架构图

二、实践环境介绍

计算机名	系统版本	IP地址	Docker版本
jeven	centos 7.6	192.168.3.166	20.10.17

三、检查本地Docker环境

1.检查本地Docker版本

检查当前系统的docker版本

[root@jeven ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:05:12 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:03:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

2.检查Docker服务状态

检查本地Docker状态是否正常

[root@jeven ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-01-02 23:06:06 CST; 2 weeks 2 days ago
     Docs: https://docs.docker.com
 Main PID: 100874 (dockerd)
    Tasks: 58
   Memory: 3.6G
   CGroup: /system.slice/docker.service

3.检查docker compose版本

检查本机docker compose版本

[root@jeven efk]# docker compose version
Docker Compose version v2.6.0

四、下载elasticsearch等相关镜像

下载elasticsearch:7.17.5镜像

[root@jeven efk]# docker pull docker.elastic.co/elasticsearch/elasticsearch:7.17.5
7.17.5: Pulling from elasticsearch/elasticsearch
5486d18d7ee8: Pull complete 
059ab60189a6: Pull complete 
f68717dc7875: Pull complete 
543411f2e134: Pull complete 
db298b0bce73: Pull complete 
841c800fd413: Pull complete 
9401277c6728: Pull complete 
d677f77adbd8: Pull complete 
f0aaff8ec792: Pull complete 
Digest: sha256:76344d5f89b13147743db0487eb76b03a7f9f0cd55abe8ab887069711f2ee27d
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.17.5
docker.elastic.co/elasticsearch/elasticsearch:7.17.5

下载dkibana:7.17.5镜像

root@jeven efk# docker pull docker.elastic.co/kibana/kibana:7.17.5
7.17.5: Pulling from kibana/kibana
5486d18d7ee8: Already exists 
16dd14f60afd: Pull complete 
d3f31853b425: Pull complete 
b189fa69d6ae: Pull complete 
84f30eda8712: Pull complete 
89732bc75041: Pull complete 
2f3ab907567d: Pull complete 
eee1465a6f1a: Pull complete 
dee78761106f: Pull complete 
77a7057d9e64: Pull complete 
4f944b4f8458: Pull complete 
0ee7e54154c9: Pull complete 
3d21668734f2: Pull complete 
2c4f2f7870ad: Pull complete 
Digest: sha256:07038507d29f21e96f3af081e4ae059661c8e16a4307776ef00d75a692cf99c7
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.17.5
docker.elastic.co/kibana/kibana:7.17.5

下载filebeat:7.17.5镜像

root@jeven efk# docker pull docker.elastic.co/beats/filebeat:7.17.5
7.17.5: Pulling from beats/filebeat
5486d18d7ee8: Already exists 
64e71d8ac435: Pull complete 
b32f71c16e16: Pull complete 
ec1697632735: Pull complete 
296368aad1f3: Pull complete 
ed91d5b07939: Pull complete 
3c3383e08710: Pull complete 
855a09516cb7: Pull complete 
bf85c6107785: Pull complete 
f2317aa670f5: Pull complete 
14c66766e716: Pull complete 
Digest: sha256:072f7c62c0d684189e9af8fc632b3235b2aceffa62a4657a1a4e0201c5976310
Status: Downloaded newer image for docker.elastic.co/beats/filebeat:7.17.5
docker.elastic.co/beats/filebeat:7.17.5

五、编辑filebeat.yaml文件

编辑filebeat.yaml配置文件

[root@jeven efk]# cat filebeat.yaml 
[root@jeven efk]# cat filebeat.yaml 
filebeat.inputs:
- type: log
  paths: 
    - '/usr/share/filebeat/logs/*'

processors:
- decode_json_fields:
    fields: ["message"]
    target: ""
    overwrite_keys: true

output.elasticsearch:
  hosts: ["http://192.168.3.166:9200"]
  indices:
    - index: "filebeat-%{+yyyy.MM.dd}"
#    - index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
#  username: '...'
#  password: '...'
setup.kibana:
  host: "http://192.168.3.166:5601"

#output.console:
#  enabled: true
#  codec.json:
#    pretty: true
#    #escape_html: false

logging.json: true
logging.metrics.enabled: false

六、部署EFK系统

1.创建数据目录

创建部署及数据目录

mkdir -p /data/efk/es/data

目录授权

chmod -R 666 /data/efk

2.编辑docker-compose.yaml文件

编辑efk系统的docker-compose.yaml部署文件

[root@jeven efk]# cat docker-compose.yaml 
version: '3.3'

services:

  elasticsearch:
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.17.5"
    container_name: elasticsearch
    restart: always
    environment:
    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    - "discovery.type=single-node"
    - "cluster.name=myes"
    - "node.name=jeven"
    # - xpack.security.enabled: "false"
    ulimits:
      memlock:
        soft: -1
        hard: -1 
    networks:
      myefk:
        ipv4_address: 172.29.120.10
        aliases:
        - es
        - jeven
    ports:
    - "9200:9200"
    - "9300:9300"
    volumes:
    - /data/efk/es/data/:/usr/share/elasticsearch/data

  kibana:
    image: "docker.elastic.co/kibana/kibana:7.17.5"
    restart: always
    environment:
      ELASTICSEARCH_URL: http://192.168.3.166:9200
      ELASTICSEARCH_HOSTS: '["http:/192.168.3.166:9200"]'
      I18N_LOCALE: zh-CN
    networks:
      myefk:
        ipv4_address: 172.29.120.20
        aliases:
          - kibana
          - kib
    ports:
    - "5601:5601"
    links:
    - "elasticsearch"

  filebeat:
    image: "docker.elastic.co/beats/filebeat:7.17.5"
    restart: always
    networks:
      myefk:
        ipv4_address: 172.29.120.30
        aliases:
          - filebeat
          - fb
    user: root
    command: ["--strict.perms=false"]
    volumes:
    - ./filebeat.yaml:/usr/share/filebeat/filebeat.yml
    - /var/lib/docker:/var/lib/docker:ro
    - /var/run/docker.sock:/var/run/docker.sock
    links:
    - "elasticsearch"
    - "kibana"


networks:
  myefk:
    driver: bridge
    ipam:
      config:
        - subnet: 172.29.120.0/24

3.部署EFK

执行docker compose up -d ，开始部署efk

[root@jeven efk]# docker compose up -d
[+] Running 4/4
 ⠿ Network efk_myefk         Created                                                                                           0.0s
 ⠿ Container elasticsearch   Started                                                                                           0.4s
 ⠿ Container efk-kibana-1    Started                                                                                           0.8s
 ⠿ Container efk-filebeat-1  Started                                                                                           0.8s

4.检查efk相关容器状态

检查efk的相关容器启动状态

[root@jeven efk]# docker compose ps
NAME                COMMAND                  SERVICE             STATUS              PORTS
efk-filebeat-1      "/usr/bin/tini -- /u…"   filebeat            running             
efk-kibana-1        "/bin/tini -- /usr/l…"   kibana              running             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp
elasticsearch       "/bin/tini -- /usr/l…"   elasticsearch       running             0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9200->9200/tcp, :::9300->9300/tcp

5.查看efk相关容器日志信息

通过docker compose logs 来查看容器运行日志信息。

[root@jeven efk]# docker compose logs |head
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"metricsEntities\" is disabled."}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","http","server","Preboot"],"pid":7,"message":"http server running at http://0.0.0.0:5601"}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format."}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \"xpack.reporting.roles.enabled\" to \"false\" to adopt the future behavior before upgrading."}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"从 8.0 开始，用户会话将在处于非活动状态 8 小时后自动超时。覆盖此值以更改超时。"}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"从 8.0 开始，将在 30 天后自动要求用户重新登录。覆盖此值以更改超时。"}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","plugins-system","standard"],"pid":7,"message":"Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]"}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","plugins","taskManager"],"pid":7,"message":"TaskManager is identified by the Kibana UUID: 5ef7fd18-4086-4ce8-8d43-91e99b733fcb"}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:48+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:48+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}

6.测试访问elasticsearch

测试本地访问elasticsearch

[root@jeven efk]# curl 192.168.3.166:9200
{
  "name" : "jeven",
  "cluster_name" : "myes",
  "cluster_uuid" : "BfacKp5xRBqNKvus7q3tIA",
  "version" : {
    "number" : "7.17.5",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",
    "build_date" : "2022-06-23T21:57:28.736740635Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}