CentOS 7下安装Logstash ELK Stack 日志管理系统(上)

介绍

The Elastic Stack - 它不是一个软件,而是Elasticsearch,Logstash,Kibana 开源软件的集合,对外是作为一个日志管理系统的开源方案。它可以从任何来源,任何格式进行日志搜索,分析获取数据,并实时进行展示。像盾牌(安全),监护者(警报)和Marvel(监测)一样为你的产品提供更多的可能。

Elasticsearch:搜索,提供分布式全文搜索引擎

Logstash: 日志收集,管理,存储

Kibana :日志的过滤web 展示 Filebeat:监控日志文件、转发

测试环境规划图

环境:ip、主机名按照如上规划,系统已经 update. 所有主机时间一致。防火墙测试环境已关闭。下面是这次elk学习的部署安装

目的:通过elk 主机收集监控主要server的系统日志、以及线上应用服务日志。

安装

3.1.基础环境检查

[root@elk ~]# hostnameelk.test.com [root@elk ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.30.67 elk.test.com192.168.30.99 rsyslog.test.com192.168.30.64 nginx.test.com

3.2.软件包

[root@elk ~]# cd elk/[root@elk elk]# wget -c https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm[root@elk elk]# wget -c https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.2-1.noarch.rpm[root@elk elk]# wget https://download.elastic.co/kibana/kibana/kibana-4.5.1-1.x86_64.rpm[root@elk elk]# wget -c https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm

3.3.检查

[root@elk elk]# lselasticsearch-2.3.3.rpm filebeat-1.2.3-x86_64.rpm kibana-4.5.1-1.x86_64.rpm logstash-2.3.2-1.noarch.rpm

服务器只需要安装e、l、k, 客户端只需要安装filebeat。

3.4.安装elasticsearch,先安装jdk,elk server 需要java 开发环境支持,由于客户端上使用的是filebeat软件,它不依赖java环境,所以不需要安装。

[root@elk elk]# yum install java-1.8.0-openjdk -y[root@elk elk]# yum localinstall elasticsearch-2.3.3.rpm -y..... Installing : elasticsearch-2.3.3-1.noarch 1/1### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service Verifying : elasticsearch-2.3.3-1.noarch 1/1Installed: elasticsearch.noarch 0:2.3.3-1[root@elk elk]# systemctl daemon-reload[root@elk elk]# systemctl enable elasticsearchCreated symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.[root@elk elk]# systemctl start elasticsearch[root@elk elk]# systemctl status elasticsearch● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2016-05-20 15:38:35 CST; 12s ago Docs: http://www.elastic.co Process: 10428 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS) Main PID: 10430 (java) CGroup: /system.slice/elasticsearch.service └─10430 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancy...May 20 15:38:38 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:38,279][INFO ][env ] [James Howlett] heap...[true]May 20 15:38:38 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:38,279][WARN ][env ] [James Howlett] max ...65536]May 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,726][INFO ][node ] [James Howlett] initializedMay 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,726][INFO ][node ] [James Howlett] starting ...May 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,915][INFO ][transport ] [James Howlett] publ...:9300}May 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,920][INFO ][discovery ] [James Howlett] elas...xx35hwMay 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,099][INFO ][cluster.service ] [James Howlett] new_...eived)May 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,164][INFO ][gateway ] [James Howlett] reco..._stateMay 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,185][INFO ][http ] [James Howlett] publ...:9200}May 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,185][INFO ][node ] [James Howlett] startedHint: Some lines were ellipsized, use -l to show in full.

检查服务

[root@elk elk]# rpm -qc elasticsearch/etc/elasticsearch/elasticsearch.yml/etc/elasticsearch/logging.yml/etc/init.d/elasticsearch/etc/sysconfig/elasticsearch/usr/lib/sysctl.d/elasticsearch.conf/usr/lib/systemd/system/elasticsearch.service/usr/lib/tmpfiles.d/elasticsearch.conf [root@elk elk]# netstat -nltp | grep java tcp6 0 0 127.0.0.1:9200 :::* LISTEN 10430/java tcp6 0 0 ::1:9200 :::* LISTEN 10430/java tcp6 0 0 127.0.0.1:9300 :::* LISTEN 10430/java tcp6 0 0 ::1:9300 :::* LISTEN 10430/java

修改防火墙,将9200、9300 端口对外开放

[root@elk elk]# firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
success
[root@elk elk]# firewall-cmd --reload
success
[root@elk elk]# firewall-cmd  --list-all
public (default, active)
  interfaces: eno16777984 eno33557248
  sources:
  services: dhcpv6-client ssh
  ports: 9200/tcp 9300/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

3.5 安装kibana

[root@elk elk]# yum localinstall kibana-4.5.1-1.x86_64.rpm –y
[root@elk elk]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
[root@elk elk]# systemctl start kibana

[root@elk elk]# systemctl status kibana
● kibana.service - no description given
   Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-05-20 15:49:02 CST; 20s ago
 Main PID: 11260 (node)
   CGroup: /system.slice/kibana.service
           └─11260 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli

May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:kbn_vi...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:markdo...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:metric...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:spyMod...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:status...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:table_...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["listening","info"],"pi...:5601"}
May 20 15:49:10 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:10+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:14 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:14+00:00","tags":["status","plugin:elasti...found"}
Hint: Some lines were ellipsized, use -l to show in full.

检查kibana服务运行(Kibana默认 进程名:node ,端口5601)

[root@elk elk]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      909/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1595/master
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      11260/node

修改防火墙,对外开放tcp/5601

[root@elk elk]# firewall-cmd --permanent --add-port=5601/tcp
Success
[root@elk elk]# firewall-cmd --reload
success
[root@elk elk]# firewall-cmd --list-all
public (default, active)
  interfaces: eno16777984 eno33557248
  sources:
  services: dhcpv6-client ssh
  ports: 9200/tcp 9300/tcp 5601/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

3.5 安装kibana

[root@elk elk]# yum localinstall kibana-4.5.1-1.x86_64.rpm –y
[root@elk elk]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
[root@elk elk]# systemctl start kibana

[root@elk elk]# systemctl status kibana
● kibana.service - no description given
   Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-05-20 15:49:02 CST; 20s ago
 Main PID: 11260 (node)
   CGroup: /system.slice/kibana.service
           └─11260 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli

May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:kbn_vi...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:markdo...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:metric...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:spyMod...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:status...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:table_...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["listening","info"],"pi...:5601"}
May 20 15:49:10 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:10+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:14 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:14+00:00","tags":["status","plugin:elasti...found"}
Hint: Some lines were ellipsized, use -l to show in full.

检查kibana服务运行(Kibana默认 进程名:node ,端口5601)

[root@elk elk]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      909/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1595/master
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      11260/node
修改防火墙,对外开放tcp/5601
[root@elk elk]# firewall-cmd --permanent --add-port=5601/tcp
Success
[root@elk elk]# firewall-cmd --reload
success
[root@elk elk]# firewall-cmd --list-all
public (default, active)
  interfaces: eno16777984 eno33557248
  sources:
  services: dhcpv6-client ssh
  ports: 9200/tcp 9300/tcp 5601/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

原文链接:http://www.cnblogs.com/hanyifeng/p/5509985.html

原文发布于微信公众号 - 马哥Linux运维(magedu-Linux)

原文发表时间:2016-08-29

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏阿杜的世界

在Spring Boot项目中使用Spock框架

Spock框架是基于Groovy语言的测试框架,Groovy与Java具备良好的互操作性,因此可以在Spring Boot项目中使用该框架写优雅、高效以及DSL...

33210
来自专栏菩提树下的杨过

CQRS框架:AxonFramework 之 Hello World

Command Query Responsibility Segregation,CQRS 这个架构好象最近博客园里讨论得比较多,有几篇园友的文章很有深度,推荐...

48270
来自专栏Python与爬虫

佛系编程[如何创建一个安全可靠的应用程序]

推荐最近在GitHub上很火的一个项目,按照介绍,你也可以创建一个安全可靠的应用程序 项目地址在>>> nocode https://github.com/k...

45290
来自专栏运维

DNS主从服务器搭建

http://blog.51cto.com/yichenyang/1911098 http://blog.51cto.com/wubinary/1379595

19110
来自专栏耕耘实录

找回win7桌面IE图标我有绝招

现在很多同学都用了微软的新一代操作系统Windows7,都为Windows7的华丽界面及更人性化得操作所深深吸引,但是由于我们大多数同学都习惯了原来的Windo...

34050
来自专栏bboysoul

使用永恒之蓝(MS17-010)漏洞入侵windows7

前几天,被勒索病毒(Wannacry)刷屏了,这个病毒是大家知道了备份文件的重要性,当然,我是不怕什么病毒的,因为我是一天备份一次文件,而且微云一份,坚果云,移...

2.8K40
来自专栏后台及大数据开发

kubernetes集群搭建(6):kubernetes基本使用演示

流程: 用户访问client应用,client应用中调用server应用,由于部署了多节点,client在访问server时应该配置server 暴露的虚拟IP...

14220
来自专栏芋道源码1024

注册中心 Eureka 源码解析 —— Eureka-Server 启动(二)之 EurekaBootStrap

本文主要基于 Eureka 1.8.X 版本 1. 概述 2. EurekaBootStrap 2.1 初始化 Eureka-Server 配置环境 2.2 初...

82440
来自专栏云计算与大数据

How to Monitor Zookeeper

As per previous articles, our general rule of thumb is “collect all possible/rea...

31840
来自专栏微信音视频小程序

教你1天搭建自己的“微视”

A simple iOS Application project is shown below to illustrate how to configure S...

1K50

扫码关注云+社区

领取腾讯云代金券