专栏首页Jed的技术阶梯009.ELK使用Redis做缓存收集Nginx日志

009.ELK使用Redis做缓存收集Nginx日志

1. 流程说明

2. 配置过程

2.1 nginx配置

log_format json  '{"time_local": "$time_local", '
                          '"remote_addr": "$remote_addr", '
                          '"referer": "$http_referer", '
                          '"request": "$request", '
                          '"status": $status, '
                          '"bytes": $body_bytes_sent, '
                          '"agent": "$http_user_agent", '
                          '"x_forwarded": "$http_x_forwarded_for", '
                          '"up_addr": "$upstream_addr", '
                          '"up_host": "$upstream_http_host", '
                          '"upstream_time": "$upstream_response_time", '
                          '"request_time": "$request_time"}';
# 使用json日志格式
access_log  /var/log/nginx/access.log main;

2.2 filebeat配置

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]
output.redis:
  hosts: ["10.0.0.104:6379"]
  keys:
    - key: "nginx_access"   
      when.contains:
        tags: "access"
    - key: "nginx_error"
      when.contains:
        tags: "error"

2.3 logstash配置

input {
  redis {
    host => "10.0.0.104"
    port => "6379"
    db => "0"
    key => "nginx_access"
    data_type => "list"
  }
  redis {
    host => "10.0.0.104"
    port => "6379"
    db => "0"
    key => "nginx_error"
    data_type => "list"
  }
}

filter {
  mutate {
    # 这两个字段转为float类型
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
    stdout {}
    if "access" in [tags] {
      elasticsearch {
        hosts => ["10.0.0.101:9200"]
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "error" in [tags] {
      elasticsearch {
        hosts => ["10.0.0.101:9200"]
        manage_template => false
        index => "nginx_error-%{+yyyy.MM}"
      }
    }
}

3. 测试

  • 启动以上服务 [root@nginx01 ~]# systemctl start nginx [root@redis01 ~]# /opt/redis/bin/redis-server /opt/redis/conf/redis_6379.conf [root@es01 ~]# systemctl start elasticsearch [root@es01 ~]# systemctl start kibana [root@nginx01 ~]# systemctl start filebeat [root@es01 ~]# /usr/share/logstash/bin/logstash -f /root/logstash.yml
  • 发送测试请求 [root@nginx01 opt]# ab -c 10 -n 1000 http://10.0.0.109:80/ [root@nginx01 opt]# ab -c 10 -n 1000 http://10.0.0.109:80/baidu
  • 查看redis 10.0.0.104:6379> keys * 1) "nginx_error" 2) "nginx_access" 10.0.0.104:6379> lpop nginx_access "{\"@timestamp\":\"2020-04-27T06:49:45.566Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"doc\",\"version\":\"6.6.0\"},\"source\":\"/var/log/nginx/access.log\",\"time_local\":\"27/Apr/2020:14:49:37 +0800\",\"x_forwarded\":\"-\",\"log\":{\"file\":{\"path\":\"/var/log/nginx/access.log\"}},\"up_addr\":\"-\",\"offset\":2775,\"request\":\"GET / HTTP/1.0\",\"request_time\":\"0.000\",\"up_host\":\"-\",\"remote_addr\":\"10.0.0.109\",\"host\":{\"name\":\"nginx01\"},\"status\":200,\"referer\":\"-\",\"tags\":[\"access\"],\"prospector\":{\"type\":\"log\"},\"input\":{\"type\":\"log\"},\"beat\":{\"name\":\"nginx01\",\"hostname\":\"nginx01\",\"version\":\"6.6.0\"},\"agent\":\"ApacheBench/2.3\",\"upstream_time\":\"-\",\"bytes\":612}" 10.0.0.104:6379> lpop nginx_error "{\"@timestamp\":\"2020-04-27T06:49:55.558Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"doc\",\"version\":\"6.6.0\"},\"log\":{\"file\":{\"path\":\"/var/log/nginx/error.log\"}},\"source\":\"/var/log/nginx/error.log\",\"tags\":[\"error\"],\"prospector\":{\"type\":\"log\"},\"input\":{\"type\":\"log\"},\"host\":{\"name\":\"nginx01\"},\"message\":\"2020/04/27 14:49:47 [error] 3031#3031: *1009 open() \\\"/usr/share/nginx/html/baidu\\\" failed (2: No such file or directory), client: 10.0.0.109, server: localhost, request: \\\"GET /baidu HTTP/1.0\\\", host: \\\"10.0.0.109\\\"\",\"offset\":1040,\"beat\":{\"version\":\"6.6.0\",\"name\":\"nginx01\",\"hostname\":\"nginx01\"}}"
  • logstash控制台输出 { "tags" => [ [0] "access" ], "request" => "GET /baidu HTTP/1.0", "offset" => 554720, "beat" => { "hostname" => "nginx01", "version" => "6.6.0", "name" => "nginx01" }, "referer" => "-", "time_local" => "27/Apr/2020:14:49:47 +0800", "input" => { "type" => "log" }, "host" => { "name" => "nginx01" }, "status" => 404, "up_addr" => "-", "up_host" => "-", "prospector" => { "type" => "log" }, "bytes" => 153, "@version" => "1", "agent" => "ApacheBench/2.3", "request_time" => 0.0, "upstream_time" => 0.0, "@timestamp" => 2020-04-27T06:49:48.731Z, "source" => "/var/log/nginx/access.log", "log" => { "file" => { "path" => "/var/log/nginx/access.log" } }, "x_forwarded" => "-", "remote_addr" => "10.0.0.109" }
  • 查看kibana GET _cat/indices yellow open nginx_access-2020.04 hikPROoJR0OK3YiX1a-ztA 5 1 2000 0 643.2kb 643.2kb yellow open nginx_error-2020.04 QUpHiZuuQSetl0m04xrMRQ 5 1 1000 0 995.9kb 995.9kb GET nginx_access-2020.04/_search { "took" : 4, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1999, "max_score" : 1.0, "hits" : [ { "_index" : "nginx_access-2020.04", "_type" : "doc", "_id" : "avBpunEBINm9vG5xGD9v", "_score" : 1.0, "_source" : { "tags" : [ "access" ], "request" : "GET / HTTP/1.0", "offset" : 246975, "time_local" : "27/Apr/2020:14:49:37 +0800", "referer" : "-", "beat" : { "hostname" : "nginx01", "version" : "6.6.0", "name" : "nginx01" }, "input" : { "type" : "log" }, "host" : { "name" : "nginx01" }, "status" : 200, "up_addr" : "-", "up_host" : "-", "prospector" : { "type" : "log" }, "bytes" : 612, "@version" : "1", "agent" : "ApacheBench/2.3", "upstream_time" : 0.0, "request_time" : 0.0, "@timestamp" : "2020-04-27T06:49:45.660Z", "source" : "/var/log/nginx/access.log", "log" : { "file" : { "path" : "/var/log/nginx/access.log" } }, "x_forwarded" : "-", "remote_addr" : "10.0.0.109" } } ] } }

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • 011.ELK使用Kafka做缓存收集Nginx日志

    CoderJed
  • 005.ELK收集Nginx日志

    ab工具用于批量发送HTTP请求到指定的URL,是一个压力测试工具,这里使用它来生成Nginx的日志

    CoderJed
  • Spark-RDD持久化

    使用不同参数的组合构造的实例被预先定义为一些值,比如MEMORY_ONLY代表着不存入磁盘,存入内存,不使用堆外内存,不进行序列化,副本数为1,使用persis...

    CoderJed
  • 解决python执行较大excel文件openpyxl慢问题

    不直接使用xlwt+xlrd是因为xlwt仅支持2003及以下版本,最大行数限制在65536,不够用,而openpyxl大概在一百多万

    砸漏
  • nginx的日志切割小知识

    感兴趣的同学可以查看此链接 http://io.upyun.com/2017/08/19/nginx-signals/

    用户2825413
  • 修改其他函数的功能的神器——python装饰器

    装饰器(Decorators)是 Python 的一个重要部分。简单地说:他们是修改其他函数的功能的函数。他们有助于让我们的代码更简短,也更Pythonic(P...

    诡途
  • Python获取时间戳

    结果如下: 1573099239000 2019-11-07 12:00:39

    hankleo
  • 《手把手教你》系列进阶篇之3-python+ selenium自动化测试 - python几种骚操作你都知道吗?(详细教程)

      这篇文章主要是给小伙伴或者童鞋们介绍和分享 python几种骚操:读取配置文件、获取根目录的相对路径、获取系统时间和格式化时间显示、字符串切割等等操作。为...

    北京-宏哥
  • django查询今天,昨天,一周,分组统计月,年

    py3study
  • python 常用资源

    mojocn

扫码关注云+社区

领取腾讯云代金券