详细图解日志收集系统Flume介绍以及安装配置

Flume的概念：flume是分布式的日志收集系统，它将各个服务器中的数据收集起来并送到指定的地方去，比如说送到图中的HDFS，简单来说flume就是收集日志的。

Flume下载下来后解压就可以使用，无需安装。

这里假设有三台服务器，其中两台是应用服务器，我把他们标识为A服务器和B服务器，还有一台就是日志服务器。

将A服务器和B服务器上的Tomcat日志全部收集到日志服务器上来，然后进行各种处理。

示例使用最常用的扇入模式（将多台机器上的日志汇集到一台机器上处理）

日志收集系统Flume安装以及常用扇入模式配置

1. 将Flume分别解压到三台服务器

2. 写配置文件

扇入流模式（将多台机器上的日志汇集到一台机器上处理）

A服务器[flume-conf.properties]

agent.sources = spoDirSrc agent.channels = memoryChannel agent.sinks = avroSink agent.sources.spoDirSrc.type = spooldir agent.sources.spoDirSrc.spoolDir = /root/logs agent.sources.spoDirSrc.channels = memoryChannel agent.sinks.avroSink.type = avro agent.sinks.avroSink.hostname = 192.168.56.186 agent.sinks.avroSink.port = 44444 agent.sinks.avroSink.channel = memoryChannel agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 1000 agent.channels.memoryChannel.transactionCapacity = 100

-----------------------------------------

B服务器[flume-conf.properties]

agent.sources = spoDirSrc agent.channels = memoryChannel agent.sinks = avroSink agent.sources.spoDirSrc.type = spooldir agent.sources.spoDirSrc.spoolDir = /root/logs agent.sources.spoDirSrc.channels = memoryChannel agent.sinks.avroSink.type = avro agent.sinks.avroSink.hostname = 192.168.56.186 agent.sinks.avroSink.port = 44444 agent.sinks.avroSink.channel = memoryChannel agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 1000 agent.channels.memoryChannel.transactionCapacity = 100

-----------------------------------------

日志服务器（192.168.56.186）[flume-conf.properties]

agent.sources = avroSrc agent.channels = memoryChannel agent.sinks = fileRollSink agent.sources.avroSrc.type = avro agent.sources.avroSrc.bind = 0.0.0.0 agent.sources.avroSrc.port = 44444 agent.sources.avroSrc.channels = memoryChannel agent.sinks.fileRollSink.type = file_roll agent.sinks.fileRollSink.sink.directory = /root/logs agent.sinks.fileRollSink.sink.rollInterval = 86400 agent.sinks.fileRollSink.channel = memoryChannel agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 1000 agent.channels.memoryChannel.transactionCapacity = 100

-----------------------------------------

3. 分别启动Flume

首先启动日志服务器上的Flume，然后再启动A和B上的Flume

> cd /usr/local/flume1.7/bin > ./flume-ng agent --conf ../conf --conf-file ../conf/flume-conf.properties --name agent -Dflume.root.logger=INFO,console &