前文传送门:如何利用NLog输出结构化日志,并在Kibana优雅分析日志?
疑问:既然应用能直接向ElasticSearch写日志,为什么我们还需要Logstash,Fluentd等日志摄取器?而且这些日志摄取器组件还成为日志收集的事实标准?
十二要素方法论
第11点
:Treat logs as event streamsA twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.
总结:您的应用不应该关注日志的路由和存储(Elasticsearch / Graylog / ...),您的日志应该只输出到stdout
,整个系统所有应用保持统一输出,由日志摄取器无侵入式收集。
在具有多种服务的dockerized环境中,每个容器都是隔离的并拥有自己的日志,我们需要一个接口来收集这些日志。 Docker Logging Driver就是干这个的:每个docker守护程序都有一个日志驱动程序,所有容器的日志都会流经该驱动程序, Docker Logging Drive让我们具备处理、转发日志的能力。
流行的库是Fluentd
, 这是一个开源的日志收集、处理、聚合组件,使用Ruby开发。
Fluent-Bit是从同一项目中fok出来的,用C写成的开源日志收集器。
Fluentd | Fluent Bit | |
---|---|---|
Scope | Containers / Servers | Containers / Servers |
Language | C & Ruby | C |
Memory | ~40MB | ~450KB |
Performance | High Performance | High Performance |
Dependencies | Built as a Ruby Gem, it requires a certain number of gems. | Zero dependencies, unless some special plugin requires them. |
Plugins | More than 650 plugins available | Around 50 plugins available |
License | Apache License v2.0 | Apache License v2.0 |
下面我们使用轻量级的Fluent-bit向ElasticSearch发送容器日志。
可通过文件或者命令行配置Fluent-Bit,下面是关键的配置节:
为收集、转发容器日志,我们需要将Fluent Bit设置为Docker Logging Driver。
forward
输入插件,监听Forward协议的转发消息es
输出插件fluent-bit.conf示例如下:
[SERVICE]
log_level info
[INPUT]
Name forward
Listen 0.0.0.0
port 24224
[OUTPUT]
Name es
Match **
Host 127.0.0.1
Port 9243
# When Logstash_Format is enabled, the Index name is composed using a prefix and the date
Logstash_Format True
# HTTP_User <user>
# HTTP_Passwd <pw>
# Alternative time key, useful if your log entries contain an @timestamp field that is used by Elasticsearch
# Time_Key es_time
# If your Elasticsearch is using TLS, configure this
# tls On
# tls.verify Off
启动ES、Fluent-Bit和一个产生日志的测试项目:
version: "3.5"
services:
elasticsearch:
image: elasticsearch:7.6.2
ports:
- "9200:9200"
environment:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.type=single-node
fluentbit:
image: fluent/fluent-bit:1.5.3
volumes:
- type: bind
source: ./fluent-bit.conf
target: /fluent-bit/etc/fluent-bit.conf
ports:
- "24224:24224"
- "24224:24224/udp"
depends_on:
- elasticsearch
ubuntu:
image: ubuntu
command: [/bin/echo, "Dotnet Plus很干,值得关注!"]
depends_on:
- fluentbit
logging:
driver: fluentd
options:
tag: docker-ubuntu
其中注意:
fluentd
作为Docker Logging Driver。检查ElasticSearch中的日志
curl localhost:9200/_cat/indices
yellow open logstash-2020.08.22 vqoyvKE4SFCcJtfo6BRmQg 1 1 1 0 6.2kb 6.2kb
curl localhost:9200/logstash-2020.08.22/_search?pretty=true&q={'matchAll':{''}}
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstash-2020.08.22",
"_type" : "_doc",
"_id" : "z0WsFnQBU8QzIbCaBXGY",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2020-08-22T14:56:33.000Z",
"log" : "Dotnet Plus很干,值得关注!",
"container_id" : "e921435eb7b8dc61bbb8e938bf67cea2694e2afd699ca71c4ef5b6d7cca12e34",
"container_name" : "/ef_ubuntu_1",
"source" : "stdout"
}
}
]
}
}
docker应用仅使用stdout,docker logging driver将日志转发至Fluent-Bit,Fluent-Bit将它们转发给Elasticsearch。
以上就是利用Fluent-Bit从容器应用收集日志并发送到ElasticSearch的基本示例。
我们再回顾下Fluent-Bit产生的背景和特性: 如今,我们环境中的信息源在不断增加,数据收集越来越复杂,需要解决
Fluent-Bit旨在成为日志收集和加工的通用瑞士军刀, 同时Fluent Bit在设计时考虑了性能和低资源消耗。