Apache Kafka的安装与配置
01
Kafka、Zookeeper安装
1. 确认系统Java版本
Kafka网络需要Java支持,版本1.8以上。
校验方法:
yjc@linux-42ti:~> java -version
Java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
(如果没有安装java , 或者版本不符合需求,可以到http://www.oracle.com/technetwork/java/javase/downloads/index.html下载rpm包,自行安装。)
2. 安装ZooKeeper
(1)下载ZooKeeper
下载地址:http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
(2)解压zookeeper-3.4.10.tar.gz,建立数据文件夹data目录
yjc@linux-42ti:/install> tar -zxf zookeeper-3.4.10.tar.gz
yjc@linux-42ti:/install> cd zookeeper-3.4.10
yjc@linux-42ti:/install/zookeeper-3.4.10> mkdir data
yjc@linux-42ti:/install/zookeeper-3.4.10> sudo chmod a+rwx bin/zkServer.sh
(3)创建配置文件zoo.cfg
yjc@linux-42ti:/install/zookeeper-3.4.10> vi conf/zoo.cfg
编辑文本如下:
tickTime=2000
dataDir=/install/zookeeper-3.4.10/data
clientPort=2181
initLimit=5
syncLimit=2
并保存。
(4)启动ZooKeeper
yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
(5)停止ZooKeeper
yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
(6)启动ZooKeeper CLI
yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkCli.sh
Connecting to localhost:2181
2018-03-01 15:56:30,701 [myid:] - INFO [main:Environment@100] -Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-03-01 15:56:30,710 [myid:] - INFO [main:Environment@100] - Client environment:host.name=
2018-03-01 15:56:30,710 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_161
......
2018-03-01 15:56:30,720 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
Welcome to ZooKeeper!
......
3. 安装Apache Kafka
(1)Kafka下载和安装
下载地址:https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.12-1.0.0.tgz
下载后解压到 /install/kafka_2.12-1.0.0
(2)启动Kafka服务器
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server.properties
......
......
[2018-03-01 16:29:07,450] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2018-03-01 16:29:07,458] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)
[2018-03-01 16:29:07,462] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
备注:
如果发现启动时报错: java.net.UnknownHostException: XXXX Name or service not known,则修改/etc/hosts文件,新增配置 127.0.0.1 XXXX
(3)停止Kafka
yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-server-stop.sh config/server.properties
02
Kafka 集群的实现
Kafka集群主要包括三种模式:
(1)单节点-单代理
(2)单节点-多代理
(3)多节点-多代理
一般而言多节点-多代理,相关节点部署在不同的Docker/虚拟机上,本文不详细展开。配置可以参考zoo.cfg。
1. 单节点-单代理
(1)启动ZooKeeper
yjc@linux-42ti:/install/zookeeper-3.4.10>cd /install/zookeeper-3.4.10
yjc@linux-42ti:/install/zookeeper-3.4.10>bin/zkServer.sh start
(2)启动Kafka
yjc@linux-42ti:/install/kafka_2.12-1.0.0> cd /install/kafka_2.12-1.0.0
yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-server-start.sh config/server.properties
(3)创建Topic(kafka-topic)
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-topics.sh --create --zookeeper
localhost:2181 --replication-factor 1 --partitions 1 --topic kafka-topic
Created topic "kafka-topic".
(4)展示Kafka Topic
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-topics.sh --list --zookeeper
localhost:2181
kafka-topic
(5)启动Producer准备产生消息
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic kafka-topic
输入相关命令后终端就可以接收相关消息了。
备注:
9092 是 /install/kafka_2.12-1.0.0/config/server.properties中的Kafka Listener配置,可修改:
listeners=PLAINTEXT://:9092
(6)启动Consumer准备接收消息
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-consumer.sh --zookeeper
localhost:2181 --topic kafka-topic --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
备注:
2181是ZooKeeper的监听端口。
/install/zookeeper-3.4.10/conf中的zoo.cfg配置,可修改:clientPort=2181
在/install/kafka_2.12-1.0.0/config/server.properties中的zookeeper.connect配置,需同步修改:zookeeper.connect=localhost:2181
(6)消息测试
2. 单节点-多代理
(1)新增配置文件 /install/kafka_2.12-1.0.0/config/server1.properties、/install/kafka_2.12-1.0.0/config/server2.properties
将server.properties 复制到server1.properties、server2.properties
同时修改以下内容:
server1.properties:
......
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
......
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
listeners=PLAINTEXT://:9093
......
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1
server2.properties:
......
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
......
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
listeners=PLAINTEXT://:9094
......
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-2
(2)启动 Kafka broker
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server1.properties
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server2.properties
(3)新建Kafka Topic
新建Topic Kafka-topic1:
yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-topics.sh --create --zookeeper
localhost:2181 --replication-factor 2 --partitions 1 --topic kafka-topic1
Created topic "kafka-topic1".
检查Topic:
yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-topics.sh --describe --zookeeper
localhost:2181
Topic:kafka-topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: kafka-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic:kafka-topic1 PartitionCount:1 ReplicationFactor:2 Configs:
Topic: kafka-topic1 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic:topic-name PartitionCount:1 ReplicationFactor:1 Configs:
Topic: topic-name Partition: 0 Leader: 0 Replicas: 0 Isr: 0
(4)启动Producer
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list
localhost:9093--topic kafka-topic1
yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list
localhost:9094--topic kafka-topic1
(5)启动Consumer
yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-console-consumer.sh --zookeeper
localhost:2181 --topic kafka-topic1 --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
(6)消息测试
END
领取专属 10元无门槛券
私享最新 技术干货