ELK的上线之路(五):—— Logstash进阶解耦

利用redis来实现logstash的解耦:

安装redis,直接采用yum安装:

vi /etc/redis.conf (简单的编辑配置)

daemonize yes(后台运行)

bind 192.168.12.74 (修改监听的地址)

启动redis:

systemctl start redis

redis-cli -h 192.168.12.74(访问测试)

编辑logstash的配置文件:

编辑:redis-out.conf(输出到redis)
启动logstash,标准输入测试。
info查看信息,并查看相关内容。
启动logstash,标准输出到elasticsearch。

生产配置文件,输入redis:

input {

file {

path => "/var/log/messages"

type => "system"

start_position => "beginning"

}

syslog {

type => "system-syslog"

host => "192.168.12.74"

port => "514"

}

file {

path => "/var/log/ucdisk/ucdisk-error.log"

type => "ucdisk-error"

start_position => "beginning"

codec => multiline {

pattern => "^[0-9]"

negate => true

what => "previous"

}

}

file {

path => "/var/log/nginx/access.log_json"

codec => "json"

type => "nginx-log"

start_position => "beginning"

}

}

output {

if [type] == "system" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

}

}

if [type] == "ucdisk-error" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

}

}

if [type] == "nginx-log" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

}

}

if [type] == "system-syslog" {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

}

}

}

生产配置文件,redis读出:

input {

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system"

type => "system"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "ucdisk-error"

type => "ucdisk-error"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "nginx-log"

type => "nginx-log"

}

redis {

host => "192.168.12.74"

port => "6379"

db => "6"

data_type => "list"

key => "system-syslog"

type => "system-syslog"

}

}

output {

if [type] == "system" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-%{+YYYY.MM.dd}"

}

}

if [type] == "ucdisk-error" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "ucdisk-error-%{+YYYY.MM.dd}"

}

}

if [type] == "nginx-log" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "nginx-log-%{+YYYY.MM.dd}"

}

}

if [type] == "system-syslog" {

elasticsearch {

hosts => [ "192.168.12.74:9200"]

index => "system-syslog-%{+YYYY.MM.dd}"

}

}

}

启动logstash:

[root@elk-node1 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed

[root@elk-node2 conf]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/redis.conf

Settings: Default filter workers: 2

Logstash startup completed

原创声明,本文系作者授权云+社区发表,未经许可,不得转载。

如有侵权,请联系 yunjia_community@tencent.com 删除。

编辑于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏Youngxj

AE博客原创 | 开源的短网址程序Myurl

1512
来自专栏木头编程 - moTzxx

Laravel+Layer 图片上传功能整理

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/u011415782/article/de...

1932
来自专栏phodal

新轮子 Mooa:使用 mooa 微服务化 Angular 应用

1332
来自专栏服务端技术杂谈

线程池监控

通过扩展线程池进行监控,通过继承线程池并重写线程池的beforeExecute,afterExecute和terminated方法,我们可以在任务执行前,执行后...

891
来自专栏铭毅天下

干货 | Elasticsearch集群黄色原因的终极探秘

绿色——最健康的状态,代表所有的主分片和副本分片都可用; 黄色——所有的主分片可用,但是部分副本分片不可用; 红色——部分主分片不可用。(此时执行查询部分数...

700
来自专栏蘑菇先生的技术笔记

topshelf和quartz内部分享

2998
来自专栏崔庆才的专栏

分布式爬虫的部署之Scrapyd-Client的使用

1653
来自专栏黑白安全

Msfvenom – Metasploit 生成 Payloads 备忘录

Metasploit 是使用最多的渗透测试框架,本文整理一些 Msfvenom 常用命令;让您快速的生成各类Payloads

581
来自专栏WindCoder

ELK实时日志管理-报错与解决

解决方案: 1. 重新安装新版本的Linux系统 2. 警告不影响使用,可以忽略

1883
来自专栏挖坑填坑

.net core + angular 项目中使用ueditor遇到的问题

这是两个问题, 1、angular中使用ueditor 2、.net core 中使用ueditor

772

扫码关注云+社区