ELK的上线之路(五):—— Logstash进阶解耦

利用redis来实现logstash的解耦:

安装redis,直接采用yum安装:

vi /etc/redis.conf (简单的编辑配置)

daemonize yes(后台运行)

bind 192.168.12.74 (修改监听的地址)

启动redis:

systemctl start redis

redis-cli -h 192.168.12.74(访问测试)

编辑logstash的配置文件:

编辑:redis-out.conf(输出到redis)
启动logstash,标准输入测试。
info查看信息,并查看相关内容。
启动logstash,标准输出到elasticsearch。

生产配置文件,输入redis:

input {

file {

path => "/var/log/messages"

type => "system"

start_position => "beginning"

}

syslog {

type => "system-syslog"

host => "192.168.12.74"

port => "514"

}

file {

path => "/var/log/ucdisk/ucdisk-error.log"

type => "ucdisk-error"

start_position => "beginning"

codec => multiline {

pattern => "^[0-9]"

negate => true

what => "previous"

}

}

file {

path => "/var/log/nginx/access.log_json"

codec => "json"

type => "nginx-log"

start_position => "beginning"

}

}

output {

if [type] == "system" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

}

}

if [type] == "ucdisk-error" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

}

}

if [type] == "nginx-log" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

}

}

if [type] == "system-syslog" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

}

}

}

生产配置文件,redis读出:

input {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

type => "system"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

type => "ucdisk-error"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

type => "nginx-log"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

type => "system-syslog"

}

}

output {

if [type] == "system" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-%{+YYYY.MM.dd}"

}

}

if [type] == "ucdisk-error" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "ucdisk-error-%{+YYYY.MM.dd}"

}

}

if [type] == "nginx-log" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "nginx-log-%{+YYYY.MM.dd}"

}

}

if [type] == "system-syslog" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-syslog-%{+YYYY.MM.dd}"

}

}

}

启动logstash:

[root@elk-node1 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed

[root@elk-node2 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed

原创声明,本文系作者授权云+社区发表,未经许可,不得转载。

如有侵权,请联系 yunjia_community@tencent.com 删除。

编辑于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏农夫安全

DiscuzX v3.4 任意文件删除漏洞

漏洞影响 DiscuzX版本 ≤ v3.4 官方于9月29日修复该漏洞: https://gitee.com/ComsenzDiscuz/DiscuzX/com...

3496
来自专栏娱乐心理测试

微信小程序获取用户信息

4.8K3
来自专栏phodal

新轮子 Mooa:使用 mooa 微服务化 Angular 应用

1672
来自专栏WindCoder

ELK实时日志管理-报错与解决

解决方案: 1. 重新安装新版本的Linux系统 2. 警告不影响使用,可以忽略

3103
来自专栏运维

Elastic Stack5.2.2升级到6.0.0注意事项

6.0版本filebeat prospectors中的document_type被禁用,原来的topic: '%{[type]}'获取文档类型的方式不可用,但是...

953
来自专栏菩提树下的杨过

ELK日志系统:Filebeat使用及Kibana如何设置登录认证

Filebeat is a lightweight, open source shipper for log file data. As the next-ge...

1121
来自专栏木头编程 - moTzxx

Laravel+Layer 图片上传功能整理

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/u011415782/article/de...

3982
来自专栏黑白安全

Msfvenom – Metasploit 生成 Payloads 备忘录

Metasploit 是使用最多的渗透测试框架,本文整理一些 Msfvenom 常用命令;让您快速的生成各类Payloads

741
来自专栏挖坑填坑

.net core + angular 项目中使用ueditor遇到的问题

这是两个问题, 1、angular中使用ueditor 2、.net core 中使用ueditor

1032
来自专栏崔庆才的专栏

分布式爬虫的部署之Scrapyd-Client的使用

1973

扫码关注云+社区