专栏首页A周立SpringCloudSpring Cloud Sleuth使用ELK收集&分析日志

Spring Cloud Sleuth使用ELK收集&分析日志

TIPS 本文基于Spring Cloud Greenwich SR2,理论兼容Spring Cloud所有版本。

应用整合

1 加依赖:

<dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-sleuth</artifactId>
    </dependency>
    <dependency>
      <groupId>net.logstash.logback</groupId>
      <artifactId>logstash-logback-encoder</artifactId>
      <version>6.1</version>
    </dependency>

注意, logstash-logback-encoder版本务必和Logback兼容,否则会导致应用启动不起来,而且不会打印任何日志!可前往 https://github.com/logstash/logstash-logback-encoder 查看和Logback的兼容性。

2 在 resources 目录下创建配置文件:logback-spring.xml ,内容如下:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <springProperty scope="context" name="springAppName" source="spring.application.name"/>
    <!-- Example for logging into the build folder of your project -->
    <property name="LOG_FILE" value="/Users/reno/Desktop/未命名文件夹/elk/logs/${springAppName}"/>

    <!-- You can override this to have a custom pattern -->
    <property name="CONSOLE_LOG_PATTERN"
              value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>

    <!-- Appender to log to console -->
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <!-- Minimum logging level to be presented in the console logs-->
            <level>DEBUG</level>
        </filter>
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <!-- Appender to log to file -->
    <appender name="flatfile" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_FILE}</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.gz</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <!-- Appender to log to file in a JSON format -->
    <appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_FILE}.json</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE}.json.%d{yyyy-MM-dd}.gz</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>UTC</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        "severity": "%level",
                        "service": "${springAppName:-}",
                        "trace": "%X{X-B3-TraceId:-}",
                        "span": "%X{X-B3-SpanId:-}",
                        "parent": "%X{X-B3-ParentSpanId:-}",
                        "exportable": "%X{X-Span-Export:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "rest": "%message"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="console"/>
        <!-- uncomment this to have also JSON logs -->
        <appender-ref ref="logstash"/>
        <!--<appender-ref ref="flatfile"/>-->
    </root>
</configuration>

3 新建 bootstrap.yml ,并将application.yml 中的以下属性移到bootstrap.yml 中。

spring:
  application:
    name: user-center

由于上面的 logback-spring.xml 含有变量(例如 springAppName ),故而 spring.application.name 属性必须设置在 bootstrap.yml 文件中,否则,logback-spring.xml 将无法正确读取属性。

测试

1 启动应用

2 日志会打印到 /Users/reno/Desktop/未命名文件夹/elk/logs/目录中 ,并且文件名称为 user-center.json ,内容类似如下:

{"@timestamp":"2019-08-29T02:38:42.468Z","severity":"DEBUG","service":"microservice-provider-user","trace":"5cf9479e966fb5ec","span":"5cf9479e966fb5ec","parent":"","exportable":"false","pid":"13144","thread":"http-nio-8000-exec-1","class":"o.s.w.s.m.m.a.RequestResponseBodyMethodProcessor","rest":"Using 'application/json;q=0.8', given [text/html, application/xhtml+xml, image/webp, image/apng, application/signed-exchange;v=b3, application/xml;q=0.9, */*;q=0.8] and supported [application/json, application/*+json, application/json, application/*+json]"}
{"@timestamp":"2019-08-29T02:38:42.469Z","severity":"DEBUG","service":"microservice-provider-user","trace":"5cf9479e966fb5ec","span":"5cf9479e966fb5ec","parent":"","exportable":"false","pid":"13144","thread":"http-nio-8000-exec-1","class":"o.s.w.s.m.m.a.RequestResponseBodyMethodProcessor","rest":"Writing [Optional[User(id=1, username=account1, name=张三, age=20, balance=100.00)]]"}
{"@timestamp":"2019-08-29T02:38:42.491Z","severity":"DEBUG","service":"microservice-provider-user","trace":"5cf9479e966fb5ec","span":"5cf9479e966fb5ec","parent":"","exportable":"false","pid":"13144","thread":"http-nio-8000-exec-1","class":"o.s.o.j.s.OpenEntityManagerInViewInterceptor","rest":"Closing JPA EntityManager in OpenEntityManagerInViewInterceptor"}
{"@timestamp":"2019-08-29T02:38:42.492Z","severity":"DEBUG","service":"microservice-provider-user","trace":"5cf9479e966fb5ec","span":"5cf9479e966fb5ec","parent":"","exportable":"false","pid":"13144","thread":"http-nio-8000-exec-1","class":"o.s.web.servlet.DispatcherServlet","rest":"Completed 200 OK"}
{"@timestamp":"2019-08-29T02:38:58.141Z","severity":"ERROR","service":"microservice-provider-user","trace":"","span":"","parent":"","exportable":"","pid":"13144","thread":"ThreadPoolTaskScheduler-1","class":"o.s.c.alibaba.nacos.discovery.NacosWatch","rest":"Error watching Nacos Service change"}

下面,只需要让Logstash收集到这个JSON文件,就可以在Kibana上检索日志啦!

ELK搭建

简单起见,本文使用Docker搭建ELK;其他搭建方式,请看官自行百度,比较简单,但很耗时。

1 创建 docker-compose.yml 文件,内容如下:

version: '3'
services:
  elasticsearch:
    image: elasticsearch:7.3.1
    environment:
      discovery.type: single-node
    ports:
      - "9200:9200"
      - "9300:9300"
  logstash:
    image: logstash:7.3.1
    command: logstash -f /etc/logstash/conf.d/logstash.conf
    volumes:
      # 挂载logstash配置文件
      - ./config:/etc/logstash/conf.d
      - /Users/reno/Desktop/未命名文件夹/elk/logs/:/opt/build/
    ports:
      - "5000:5000"
  kibana:
    image: kibana:7.3.1
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    ports:
      - "5601:5601"

需要注意,上面的 /Users/reno/Desktop/未命名文件夹/elk/logs/ 需要改成你应用的打印路径。

2 在docker-compose.yml文件所在目录创建 config/logstash.conf ,内容如下:

input {
  file {
    codec => json
    path => "/opt/build/*.json"  # 改成你项目打印的json日志文件。
  }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
  }
}
output {
  elasticsearch {
    hosts => "elasticsearch:9200"  # 改成你的Elasticsearch地址
  }
}

3 启动ELK

docker-compose up

测试Sleuth & ELK

1 访问你微服务的API,让它生成一些日志(如果产生日志比较少,可将 org.springframework 包的日志级别设为 debug

2 访问 http://localhost:5601 (Kibana地址),可看到类似如下的界面,按照如图配置Kibana。

3 输入条件,即可分析日志:

原理分析

原理比较简单:

•让Sleuth打印JSON格式的日志;•然后在Logstash的配置文件中,配置grok语法,解析并收集JSON格式的日志,并存储到Elasticsearch中去;•Kibana可视化分析日志。

本文分享自微信公众号 - IT牧场(itmuch_com),作者:itmuch

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。

原始发表时间:2019-09-01

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • 跟我学Spring Cloud(Finchley版)-09-Feign

    经过前文讲解,我们已使用Eureka实现服务发现;使用Ribbon实现了负载均衡这种听起来很高端的东西。我们的架构已经初具雏形,但依然存在很多问题,下面不妨来分...

    用户1516716
  • 高级 Java 程序猿必须掌握的 17 个 JVM 参数

    cnblogs.com/Simple-Object/p/10272326.html

    用户1516716
  • 跟我学Spring Cloud(Finchley版)-13-通用方式使用Hystrix

    Hystrix是由Netflix开源的一个延迟和容错库,用于隔离访问远程系统、服务或者第三方库,防止级联失败,从而提升系统的可用性与容错性。Hystrix主要通...

    用户1516716
  • 金黄葡萄球菌RNA-seq数据分析

    这里出现问题了,突变株的比对率太低,不到1%,这是不可能的,怀疑样品污染,然后随机挑选了5条序列blast了下,发现应该是被溶血葡萄球菌污染。

    Y大宽
  • PHP面向对象核心(二)——继承、多态、接口

    PHP面向对象核心(二) (原创内容,转载请注明来源,谢谢) 三、继承与多态 3.1 继承 1、继承是类级别的复用,关键词为extends;多态是方法级别的复用...

    用户1327360
  • flask第二十二篇——模板【4】过滤器

    用户2149234
  • 韩大学设立人工智能武器实验室,遭50位顶尖学者抵制

    韩国著名大学——韩国科学技术院(Korea Advanced Institute of Science andTechnology,KAIST)于2018年2月...

    人工智能快报
  • 风险量化分析案例:投标与报价最优化建模分析

    项目或者设备得供应商投标价格得方法有很多。一种常见得方法是:首先估计项目或设备得成本基值,然后确定投标价格再成本基值得基础上得提高比例,即提价比例,最后形成投标...

    沉默的白面书生
  • 数字图像处理界标准图像 Lena 后面的故事

      熟悉图像处理或者压缩的工程师、研究人员和学生,经常在他们的实验或者项目任务里使用“Lenna”或者“Lena”的图像。Lenna 图像已经成为被广泛使用的测...

    NingHeChuan
  • 学Python必背的初级单词,你都背了吗?

    argument 引数(传给函式的值)。叁见 parameter 叁数、实质叁数、实叁、自变量

    python学习教程

扫码关注云+社区

领取腾讯云代金券