flume-ng tmp

flume-ng 是一个分布式，高可用的日志收集系统。主要用来将分布在不同服务器上的业务日志汇总在一个集中的数据存储中心

一 安装与环境配置

下载地址 http://flume.apache.org/download.html ， 下载Apache Flume binary至目标服务器解压

运行环境java版本：Java 1.6 or later (Java 1.7 Recommended)

配置JAVA_HOME变量

将解压文件路径/bin配置加入环境变量

二 命令参数

Usage: /home/dongxiao.yang/apache-flume-1.4.0-bin/bin/flume-ng <command> [options]...

commands: help display this help text agent run a Flume agent avro-client run an avro Flume client version show Flume version info

global options: --conf,-c <conf> use configs in <conf> directory --classpath,-C <cp> append to the classpath --dryrun,-d do not actually start Flume, just print the command --plugins-path <dirs> colon-separated list of plugins.d directories. See the plugins.d section in the user guide for more details. Default: $FLUME_HOME/plugins.d -Dproperty=value sets a Java system property value -Xproperty=value sets a Java -X option

agent options: --conf-file,-f <file> specify a config file (required) --name,-n <name> the name of this agent (required) --help,-h display help text

avro-client options: --rpcProps,-P <file> RPC client properties file with server connection params --host,-H <host> hostname to which events will be sent --port,-p <port> port of the avro source --dirname <dir> directory to stream to avro source --filename,-F <file> text file to stream to avro source (default: std input) --headerFile,-R <file> File containing event headers as key/value pairs on each new line --help,-h display help text

Either --rpcProps or both --host and --port must be specified.

Note that if <conf> directory is specified, then it is always included first in the classpath.

配置文件简单例子

#define agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 sink2

#Describe the source agent1.sources.source1.type = exec agent1.sources.source1.command = tail -F /srv/apps/taskworker/log/taskworker.log agent1.sources.source1.interceptors=e1 agent1.sources.source1.interceptors.e1.type=timestamp

#Describe the sink agent1.sinks.sink1.type = avro agent1.sinks.sink1.hostname= 10.4.1.100 agent1.sinks.sink1.port = 10000

#Describe the channnel agent1.channels.channel1.type = file agent1.channels.channel1.checkpointDir = /home/dongxiao.yang/checkpoint agent1.channels.channel1.dataDirs = /home/dongxiao.yang/data

#Bind the source and sink to the channel agent1.sources.source1.channels = channel1 agent1.sinks.sink1.channel = channel1 agent1.sinks.sink2.channel = channel1

启动命令格式：

flume-ng agent --conf /home/dongxiao.yang/apache-flume-1.4.0-bin/conf/ --conf-file /home/dongxiao.yang/apache-flume-1.4.0-bin/conf/

--name agent1 -Dflume.root.logger=INFO,console -Duser.timezone=UTC

参考资料：http://flume.apache.org/FlumeUserGuide.html 官方文档

Apache Flume Distributed Log Collection for Hadoop.pdf 基于1.3版本，主要介绍了收集常见日志文件写入hdfs的几个结构