Flume + Kafka整合

sparkle123

发布于 2018-04-26 13:29:43

7030

发布于 2018-04-26 13:29:43

文章被收录于专栏：大数据-Hadoop、Spark 大数据-Hadoop、Spark大数据-Hadoop、Spark

Flume的安装与综合使用

https://cloud.tencent.com/developer/article/1109430

实时日志采集框架图

Flume + Kafka整合.png

1.在$FLUME_HOME/conf下新增一个配置文件avro-memory-kafka.conf

avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel

avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = 192.168.21.110
avro-memory-kafka.sources.avro-source.port = 44444

avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = 192.168.21.110:9092
avro-memory-kafka.sinks.kafka-sink.topic = hello_topic
avro-memory-kafka.sinks.kafka-sink.batchSize = 5
avro-memory-kafka.sinks.kafka-sink.requiredAcks =1

avro-memory-kafka.channels.memory-channel.type = memory

avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

2.重点是配置kafka sink的内容，参考flume-ng-1.6.0-cdh5.7.0官方的文档:

kafka sink.png

3.先启动avro-memory-kafka监听

flume-ng agent \
--name avro-memory-kafka \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/avro-memory-kafka.conf \
-Dflume.root.logger=INFO,console

4.然后启动exec-memory-avro.conf

flume-ng agent \
--name exec-memory-avro \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
-Dflume.root.logger=INFO,console

5.启动kafka-console-consumer监听kafka的消息变化

kafka-console-consumer.sh --zookeeper 192.168.21.110:2181 --topic hello_topic

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

原始发表：2018.04.12 ，如有侵权请联系 cloudcommunity@tencent.com 删除

kafka

本文分享自作者个人站点/博客前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体分享计划，欢迎热爱写作的你一起参与！

kafka

登录后参与评论

0 条评论

热度

Flume + Kafka整合

Flume + Kafka整合

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐