ELK的上线之路（五）：—— Logstash进阶解耦

利用redis来实现logstash的解耦：

安装redis,直接采用yum安装：

vi /etc/redis.conf (简单的编辑配置)

daemonize yes（后台运行）

bind 192.168.12.74 （修改监听的地址）

启动redis:

systemctl start redis

redis-cli -h 192.168.12.74（访问测试）

编辑logstash的配置文件：

编辑：redis-out.conf(输出到redis)

启动logstash,标准输入测试。

info查看信息，并查看相关内容。

启动logstash,标准输出到elasticsearch。

生产配置文件,输入redis:

input {

file {

path => "/var/log/messages"

type => "system"

start_position => "beginning"

}

syslog {

type => "system-syslog"

host => "192.168.12.74"

port => "514"

}

file {

path => "/var/log/ucdisk/ucdisk-error.log"

type => "ucdisk-error"

start_position => "beginning"

codec => multiline {

pattern => "^[0-9]"

negate => true

what => "previous"

}

}

file {

path => "/var/log/nginx/access.log_json"

codec => "json"

type => "nginx-log"

start_position => "beginning"

}

}

output {

if [type] == "system" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

}

}

if [type] == "ucdisk-error" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

}

}

if [type] == "nginx-log" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

}

}

if [type] == "system-syslog" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

}

}

}

生产配置文件,redis读出:

input {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

type => "system"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

type => "ucdisk-error"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

type => "nginx-log"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

type => "system-syslog"

}

}

output {

if [type] == "system" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-%{+YYYY.MM.dd}"

}

}

if [type] == "ucdisk-error" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "ucdisk-error-%{+YYYY.MM.dd}"

}

}

if [type] == "nginx-log" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "nginx-log-%{+YYYY.MM.dd}"

}

}

if [type] == "system-syslog" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-syslog-%{+YYYY.MM.dd}"

}

}

}

启动logstash：

[root@elk-node1 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed

[root@elk-node2 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed