Log system architecture

0. 技术选型参考

1. Collector

Keywords: Collector, Processor

名称

Beats

Fluentd-bit

Introduction

Beats are a collector and processor of lightweight (resource efficient, no dependencies, small) and open source log shippers that act as agents installed on the different servers in your infrastructure for collecting logs or metrics.

Fluent Bit was born to address the need for a high performance and optimized tool that can collect and process data from any input source, unify that data and deliver it to multiple destinations.

Owner

Elastic

Treasure Data

Open Source

True

True

Github Stars

5742

608

License

Apache License v2.0

Apache License v2.0

Scope

Containers / Servers / K8S

Containers / Servers / K8S

Language

Go

C

Memory

~10MB

~500KB

Performance

High

High

Dependencies

Zero dependencies, unless some special plugin requires them.

Zero dependencies, unless some special plugin requires them.

Category

Auditbeat,Filebeat,Heartbeat,Metricbeat,Packetbeat,Winlogbeat

NaN

Configuration

File(.yml)/Cmd

File(custom file extension and syntax)/Cmd

Essence

Collector & Processor

Collector & Processor

Input/Module

File, Docker, Syslog, Nginx, Mysql, Postgresql, etc

File,CPU, Disk, Docker, Syslog, etc

Output

Elasticsearch, Logstash, Kafka, Redis, File, Console

ES, File, Kafka, etc

1.1 Filebeat 架构图

  1. Ingest Node - A es plugin which pre-process documents before the actual document indexing happen and replace for Logstash. The ingest node intercepts bulk and index requests, it applies transformations, and it then passes the documents back to the index or bulk APIs. Define a pipeline(Processors) that specifies a series of processors, then register the pipeline id in Filebeat configuration file.
  2. Kafka - Prevent loss of data and manage logging output speed.

1.2 Fluent bit 架构图

Name

Description

Samples

Input

Entry point of data. Implemented through Input Plugins, this interface allows to gather or receive data.

Samples

Parser

Parsers allow to convert unstructured data gathered from the Input interface into a structured one. Parsers are optional and depends on Input plugins.

Prospector and processors in Filebeat

Filter

The filtering mechanism allows to alter the data ingested by the Input plugins. Filters are implemented as plugins.

Prospector and processors in Filebeat

Buffer

By default, the data ingested by the Input plugins, resides in memory until is routed and delivered to an Output interface.

Routing

Data ingested by an Input interface is tagged, that means that a Tag is assigned and this one is used to determinate where the data should be routed based on a match rule.

Output

An output defines a destination for the data. Destinations are handled by output plugins. Note that thanks to the Routing interface, the data can be delivered to multiple destinations.

Samples

2. Log Transporter

Keywords: Collector, Processor, Aggregator

名称

Logstah

Fluentd

Introduction

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your stash.

Fluentd is an open source data collector, which lets you unify the data.

Owner

Elastic

Treasure Data

Open Source

True

True

Github Stars

9105

6489

License

Apache License v2.0

Apache License v2.0

Scope

Containers / Servers / K8S

Containers / Servers / K8S

Language

JRuby(JVM)

Ruby & C

Memory

200MB+

~40MB

Performance

Middle

High

Dependencies

JVM

Ruby Gem

Configuration

File(custom file extension and syntax)/Cmd

File(custom file extension and syntax)/Cmd

Essence

Collector, Processor, Aggregator

CCollector, Processor, Aggregator

Input/Module

Limited only by your imagination(Serilog)

Limited only by your imagination(Nlog)

Output

Limited only by your imagination

Limited only by your imagination

Further Reading: Fluentd vs. Logstash: A Comparison of Log Collectors

3. 初步总结

比较

Beats + Logstash

Fluentd bit + Fluentd

说明

功能实现

基本一致

安装与配置简易性

内存占用

JVM 特性使然

可靠性

前者使用 registry file + redis 实现可靠性,后者使用内置 buffering 实现可靠性

可扩展性

插件生态和可扩展性基本一致。后者为分布型插件管理

趋势

ELK -> EFK

其他

前者更倾向于使用 go & java 技术栈,后者有 docker, k8s 官方 log driver 类型和案例支持

Tips: 任一层级都可以自由替换.

4. Visualizer

Keywords: Query, Analyze, Monitor

名称

Kibana

Grafana

Introduction

Kibana is an open source data visualization plugin for Elasticsearch.

Data visualization & Monitoring with support for Graphite, InfluxDB, Prometheus, Elasticsearch and many more databases.The leading open source software for time series analytics.

Owner

Elastic

Grafana

Open Source

True

True

Github Stars

9k+

22k+

License

Apache License v2.0

Apache License v2.0

Scope

ElasticSearch only

ElasticSearch, InfluxDB, PostgreSQL etc

Language

Javascript

Go & Typescript

Configuration

File(.yml)/Cmd

File(custom file extension and syntax)/Cmd

Simple Query

Lucene syntax and filter components

filter components.Different from each other data source

Full-Text Query

Yes

No

Security

Plugins or libraries

Integration

Notification

Plugins or libraries

Integration

Advantages

Log, ES

Multiple data source, APM, Timeseries

Working together.

5. Log Storage and Analyzer

Keywords:Storage, ES, Postgresql, Zombodb, Arangodb

5.1 ElasticSearch

  1. 同时支持单文档的对象搜索+模糊搜索+全文搜索
  2. Skywalking 官方支持存储媒介
  3. 作为流行 Output 支持绝大部分 Log 相关系统
  4. 天生分布式
  5. 一键设置过期窗口,索引重建
  6. ……

  1. 占用资源较多,对存储介质要求高
  2. 运维成本更高
  3. 持久化
  4. 安全性 - Search Guard
  5. ……

6. 总结

  1. Sinks(Log sinks, Beats, Fluentd-bit) -> Storages(ElasticSearch, Postgresql,Zombodb etc).
  2. Collctors(Beats, Fluentd-bit) -> Kafka -> Fluentd -> Storages(ElasticSearch, Postgresql,Zombodb etc).

7. 扩展

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏杨建荣的学习笔记

一个Oracle bug的手工修复(r6笔记第59天)

在上周五的时候,本来一个例行巡检,想扩充一些表空间,结果弄巧成拙,因为一个drop datafile的操作直接导致了一主两备的两个备库MRP直接抛出了ORA-6...

2895
来自专栏bboysoul

使用永恒之蓝(MS17-010)漏洞入侵windows7

前几天,被勒索病毒(Wannacry)刷屏了,这个病毒是大家知道了备份文件的重要性,当然,我是不怕什么病毒的,因为我是一天备份一次文件,而且微云一份,坚果云,移...

2.2K4
来自专栏耕耘实录

找回win7桌面IE图标我有绝招

现在很多同学都用了微软的新一代操作系统Windows7,都为Windows7的华丽界面及更人性化得操作所深深吸引,但是由于我们大多数同学都习惯了原来的Windo...

3105
来自专栏Android先生

【漫画技术】Android跨进程通信

Tips:4个环节,共计约9小时的精心打磨完成上线,同时也非常感谢参与审稿的同学。

1172
来自专栏Samego开发资源

It can make your ssh login simply as well as efficiently on Mac or Linux

It can make your ssh login simply as well as efficiently on Mac or LInux. 点我翻译 ...

952
来自专栏技术博文

Linux命令英文全称

su:Swith user  切换用户,切换到root用户 cat: Concatenate  串联 uname: Unix name  系统名称 df: Di...

3975
来自专栏蓝天

Redis模块开发示例

实现一个Redis module,支持两个扩展命令: 1) 可同时对hash的多个field进行incr操作; 2) incrby同时设置一个key的过期时...

1183
来自专栏Android先生

Dagger2神器入门(二)

在Dagger2神器入门(一)中,我们了解了什么是依赖注入,那么在这一章中,我们将逐渐入门Dagger2。接下来我们会通过demo形式给大家展现Dagger2的...

1003
来自专栏菩提树下的杨过

CQRS框架:AxonFramework 之 Hello World

Command Query Responsibility Segregation,CQRS 这个架构好象最近博客园里讨论得比较多,有几篇园友的文章很有深度,推荐...

4507
来自专栏数据库新发现

Oracle数据库恢复:归档日志损坏案例一则

链接:http://www.eygle.com/archives/2010/11/recover_archivelog_corruption.html

1022

扫码关注云+社区

领取腾讯云代金券