ELK的上线之路(五):—— Logstash进阶解耦

利用redis来实现logstash的解耦:

安装redis,直接采用yum安装:

vi /etc/redis.conf (简单的编辑配置)

daemonize yes(后台运行)

bind 192.168.12.74 (修改监听的地址)

启动redis:

systemctl start redis

redis-cli -h 192.168.12.74(访问测试)

编辑logstash的配置文件:

编辑:redis-out.conf(输出到redis)
启动logstash,标准输入测试。
info查看信息,并查看相关内容。
启动logstash,标准输出到elasticsearch。

生产配置文件,输入redis:

input {

file {

path => "/var/log/messages"

type => "system"

start_position => "beginning"

}

syslog {

type => "system-syslog"

host => "192.168.12.74"

port => "514"

}

file {

path => "/var/log/ucdisk/ucdisk-error.log"

type => "ucdisk-error"

start_position => "beginning"

codec => multiline {

pattern => "^[0-9]"

negate => true

what => "previous"

}

}

file {

path => "/var/log/nginx/access.log_json"

codec => "json"

type => "nginx-log"

start_position => "beginning"

}

}

output {

if [type] == "system" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

}

}

if [type] == "ucdisk-error" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

}

}

if [type] == "nginx-log" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

}

}

if [type] == "system-syslog" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

}

}

}

生产配置文件,redis读出:

input {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

type => "system"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

type => "ucdisk-error"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

type => "nginx-log"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

type => "system-syslog"

}

}

output {

if [type] == "system" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-%{+YYYY.MM.dd}"

}

}

if [type] == "ucdisk-error" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "ucdisk-error-%{+YYYY.MM.dd}"

}

}

if [type] == "nginx-log" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "nginx-log-%{+YYYY.MM.dd}"

}

}

if [type] == "system-syslog" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-syslog-%{+YYYY.MM.dd}"

}

}

}

启动logstash:

[root@elk-node1 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed

[root@elk-node2 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed

原创声明,本文系作者授权云+社区发表,未经许可,不得转载。

如有侵权,请联系 yunjia_community@tencent.com 删除。

编辑于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏贾鹏辉的技术专栏@CrazyCodeBoy

轻松搞定GitHub Pages

GitHub支持创建个人或组织以及项目这两种类型的网站。 本文章将向大家分享如何为项目、组织或个人创建一个GitHub Pages。 为项目创建GitHub ...

3498
来自专栏Jackson0714

Android Studio 导入External Libraries

2649
来自专栏鬼谷君

kvm解决1000M网卡问题

2473
来自专栏GreenLeaves

tomcat

一  tomcat安装使用: 1.(前提条件)安装JDK,配置Java的环境变量. 2.tomcat分为安装版和解压版  安装版(一般给客户使用)  安装版默认...

2998
来自专栏计算机编程

Jenkins mac上配置ionic android

4.1、下载android studio AndroidStudio 官网:<a href="https://developer.android.google...

891
来自专栏狂码一生

Linux服务器下搭建Apache+MySql+PHP+phpMyAdmin运行环境

一、安装Apache   1、检查系统是否安装Apache   rpm -qa | grep httpd   2、安装   yum -y install htt...

4228
来自专栏blog

利用Hugo和Github Pages免费创建并永久托管网站

Hugo可以让你轻松生成静态网站,比如个人博客、API文档、公司主页等,你只需要提供markdown格式的文本,它就能帮你渲染成各种你想要的样式,只需要安装想...

1391
来自专栏Android相关

Ijkplayer编译

ijkplayer最好用ndk-r10e来进行编译,否则会出现很多Cmake等等问题

633
来自专栏康怀帅的专栏

macOS aria2 命令行使用详解

aria2 是一个下载工具。 aria2 https://github.com/aria2/aria2 BaiduExporter https://gith...

3996
来自专栏无所事事者爱嘲笑

mac上adb command not found

1694

扫码关注云+社区