Apache Kafka的安装与配置（二）

Apache Kafka的安装与配置

01

Kafka、Zookeeper安装

1. 确认系统Java版本

Kafka网络需要Java支持，版本1.8以上。

校验方法：

yjc@linux-42ti:~> java -version

Java version "1.8.0_161"

Java(TM) SE Runtime Environment (build 1.8.0_161-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

（如果没有安装java , 或者版本不符合需求，可以到http://www.oracle.com/technetwork/java/javase/downloads/index.html下载rpm包，自行安装。）

2. 安装ZooKeeper

（1）下载ZooKeeper

下载地址：http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

（2）解压zookeeper-3.4.10.tar.gz，建立数据文件夹data目录

yjc@linux-42ti:/install> tar -zxf zookeeper-3.4.10.tar.gz

yjc@linux-42ti:/install> cd zookeeper-3.4.10

yjc@linux-42ti:/install/zookeeper-3.4.10> mkdir data

yjc@linux-42ti:/install/zookeeper-3.4.10> sudo chmod a+rwx bin/zkServer.sh

（3）创建配置文件zoo.cfg

yjc@linux-42ti:/install/zookeeper-3.4.10> vi conf/zoo.cfg

编辑文本如下：

tickTime=2000

dataDir=/install/zookeeper-3.4.10/data

clientPort=2181

initLimit=5

syncLimit=2

并保存。

（4）启动ZooKeeper

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

（5）停止ZooKeeper

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh stop

ZooKeeper JMX enabled by default

Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg

Stopping zookeeper ... STOPPED

（6）启动ZooKeeper CLI

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkCli.sh

Connecting to localhost:2181

2018-03-01 15:56:30,701 [myid:] - INFO [main:Environment@100] -Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT

2018-03-01 15:56:30,710 [myid:] - INFO [main:Environment@100] - Client environment:host.name=

2018-03-01 15:56:30,710 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_161

......

2018-03-01 15:56:30,720 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e

Welcome to ZooKeeper!

......

3. 安装Apache Kafka

（1）Kafka下载和安装

下载地址：https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.12-1.0.0.tgz

下载后解压到 /install/kafka_2.12-1.0.0

（2）启动Kafka服务器

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server.properties

......

......

[2018-03-01 16:29:07,450] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)

[2018-03-01 16:29:07,458] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)

[2018-03-01 16:29:07,462] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

备注：

如果发现启动时报错： java.net.UnknownHostException: XXXX Name or service not known，则修改/etc/hosts文件，新增配置 127.0.0.1 XXXX

（3）停止Kafka

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-server-stop.sh config/server.properties

02

Kafka 集群的实现

Kafka集群主要包括三种模式：

（1）单节点-单代理

（2）单节点-多代理

（3）多节点-多代理

一般而言多节点-多代理，相关节点部署在不同的Docker/虚拟机上，本文不详细展开。配置可以参考zoo.cfg。

1. 单节点-单代理

（1）启动ZooKeeper

yjc@linux-42ti:/install/zookeeper-3.4.10>cd /install/zookeeper-3.4.10

yjc@linux-42ti:/install/zookeeper-3.4.10>bin/zkServer.sh start

（2）启动Kafka

yjc@linux-42ti:/install/kafka_2.12-1.0.0> cd /install/kafka_2.12-1.0.0

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-server-start.sh config/server.properties

（3）创建Topic（kafka-topic）

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-topics.sh --create --zookeeper

localhost:2181 --replication-factor 1 --partitions 1 --topic kafka-topic

Created topic "kafka-topic".

（4）展示Kafka Topic

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-topics.sh --list --zookeeper

localhost:2181

kafka-topic

（5）启动Producer准备产生消息

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list

localhost:9092 --topic kafka-topic

输入相关命令后终端就可以接收相关消息了。

备注：

9092 是 /install/kafka_2.12-1.0.0/config/server.properties中的Kafka Listener配置，可修改：

listeners=PLAINTEXT://:9092

（6）启动Consumer准备接收消息

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-consumer.sh --zookeeper

localhost:2181 --topic kafka-topic --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

备注：

2181是ZooKeeper的监听端口。

/install/zookeeper-3.4.10/conf中的zoo.cfg配置，可修改：clientPort=2181

在/install/kafka_2.12-1.0.0/config/server.properties中的zookeeper.connect配置，需同步修改：zookeeper.connect=localhost:2181

（6）消息测试

2. 单节点-多代理

（1）新增配置文件 /install/kafka_2.12-1.0.0/config/server1.properties、/install/kafka_2.12-1.0.0/config/server2.properties

将server.properties 复制到server1.properties、server2.properties

同时修改以下内容：

server1.properties：

......

# The id of the broker. This must be set to a unique integer for each broker.

broker.id=1

......

# listeners = PLAINTEXT://your.host.name:9092

#listeners=PLAINTEXT://:9092

listeners=PLAINTEXT://:9093

......

# A comma seperated list of directories under which to store log files

log.dirs=/tmp/kafka-logs-1

server2.properties：

......

# The id of the broker. This must be set to a unique integer for each broker.

broker.id=2

......

# listeners = PLAINTEXT://your.host.name:9092

#listeners=PLAINTEXT://:9092

listeners=PLAINTEXT://:9094

......

# A comma seperated list of directories under which to store log files

log.dirs=/tmp/kafka-logs-2

（2）启动 Kafka broker

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server1.properties

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server2.properties

（3）新建Kafka Topic

新建Topic Kafka-topic1：

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-topics.sh --create --zookeeper

localhost:2181 --replication-factor 2 --partitions 1 --topic kafka-topic1

Created topic "kafka-topic1".

检查Topic：

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-topics.sh --describe --zookeeper

localhost:2181

Topic:kafka-topic PartitionCount:1 ReplicationFactor:1 Configs:

Topic: kafka-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0

Topic:kafka-topic1 PartitionCount:1 ReplicationFactor:2 Configs:

Topic: kafka-topic1 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2

Topic:topic-name PartitionCount:1 ReplicationFactor:1 Configs:

Topic: topic-name Partition: 0 Leader: 0 Replicas: 0 Isr: 0

（4）启动Producer

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list

localhost:9093--topic kafka-topic1

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list

localhost:9094--topic kafka-topic1

（5）启动Consumer

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-console-consumer.sh --zookeeper

localhost:2181 --topic kafka-topic1 --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

（6）消息测试

END