Apache Kafka的安装与配置(二)

Apache Kafka的安装与配置

01

Kafka、Zookeeper安装

1. 确认系统Java版本

Kafka网络需要Java支持,版本1.8以上。

校验方法:

yjc@linux-42ti:~> java -version

Java version "1.8.0_161"

Java(TM) SE Runtime Environment (build 1.8.0_161-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

(如果没有安装java , 或者版本不符合需求,可以到http://www.oracle.com/technetwork/java/javase/downloads/index.html下载rpm包,自行安装。)

2. 安装ZooKeeper

(1)下载ZooKeeper

下载地址:http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

(2)解压zookeeper-3.4.10.tar.gz,建立数据文件夹data目录

yjc@linux-42ti:/install> tar -zxf zookeeper-3.4.10.tar.gz

yjc@linux-42ti:/install> cd zookeeper-3.4.10

yjc@linux-42ti:/install/zookeeper-3.4.10> mkdir data

yjc@linux-42ti:/install/zookeeper-3.4.10> sudo chmod a+rwx bin/zkServer.sh

(3)创建配置文件zoo.cfg

yjc@linux-42ti:/install/zookeeper-3.4.10> vi conf/zoo.cfg

编辑文本如下:

tickTime=2000

dataDir=/install/zookeeper-3.4.10/data

clientPort=2181

initLimit=5

syncLimit=2

并保存。

(4)启动ZooKeeper

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

(5)停止ZooKeeper

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh stop

ZooKeeper JMX enabled by default

Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg

Stopping zookeeper ... STOPPED

(6)启动ZooKeeper CLI

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /install/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

yjc@linux-42ti:/install/zookeeper-3.4.10> bin/zkCli.sh

Connecting to localhost:2181

2018-03-01 15:56:30,701 [myid:] - INFO [main:Environment@100] -Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT

2018-03-01 15:56:30,710 [myid:] - INFO [main:Environment@100] - Client environment:host.name=

2018-03-01 15:56:30,710 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_161

......

2018-03-01 15:56:30,720 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e

Welcome to ZooKeeper!

......

3. 安装Apache Kafka

(1)Kafka下载和安装

下载地址:https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.12-1.0.0.tgz

下载后解压到 /install/kafka_2.12-1.0.0

(2)启动Kafka服务器

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server.properties

......

......

[2018-03-01 16:29:07,450] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)

[2018-03-01 16:29:07,458] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)

[2018-03-01 16:29:07,462] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

备注:

如果发现启动时报错: java.net.UnknownHostException: XXXX Name or service not known,则修改/etc/hosts文件,新增配置 127.0.0.1 XXXX

(3)停止Kafka

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-server-stop.sh config/server.properties

02

Kafka 集群的实现

Kafka集群主要包括三种模式:

(1)单节点-单代理

(2)单节点-多代理

(3)多节点-多代理

一般而言多节点-多代理,相关节点部署在不同的Docker/虚拟机上,本文不详细展开。配置可以参考zoo.cfg。

1. 单节点-单代理

(1)启动ZooKeeper

yjc@linux-42ti:/install/zookeeper-3.4.10>cd /install/zookeeper-3.4.10

yjc@linux-42ti:/install/zookeeper-3.4.10>bin/zkServer.sh start

(2)启动Kafka

yjc@linux-42ti:/install/kafka_2.12-1.0.0> cd /install/kafka_2.12-1.0.0

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-server-start.sh config/server.properties

(3)创建Topic(kafka-topic)

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-topics.sh --create --zookeeper

localhost:2181 --replication-factor 1 --partitions 1 --topic kafka-topic

Created topic "kafka-topic".

(4)展示Kafka Topic

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-topics.sh --list --zookeeper

localhost:2181

kafka-topic

(5)启动Producer准备产生消息

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list

localhost:9092 --topic kafka-topic

输入相关命令后终端就可以接收相关消息了。

备注:

9092 是 /install/kafka_2.12-1.0.0/config/server.properties中的Kafka Listener配置,可修改:

listeners=PLAINTEXT://:9092

(6)启动Consumer准备接收消息

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-consumer.sh --zookeeper

localhost:2181 --topic kafka-topic --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

备注:

2181是ZooKeeper的监听端口。

/install/zookeeper-3.4.10/conf中的zoo.cfg配置,可修改:clientPort=2181

在/install/kafka_2.12-1.0.0/config/server.properties中的zookeeper.connect配置,需同步修改:zookeeper.connect=localhost:2181

(6)消息测试

2. 单节点-多代理

(1)新增配置文件 /install/kafka_2.12-1.0.0/config/server1.properties、/install/kafka_2.12-1.0.0/config/server2.properties

将server.properties 复制到server1.properties、server2.properties

同时修改以下内容:

server1.properties:

......

# The id of the broker. This must be set to a unique integer for each broker.

broker.id=1

......

# listeners = PLAINTEXT://your.host.name:9092

#listeners=PLAINTEXT://:9092

listeners=PLAINTEXT://:9093

......

# A comma seperated list of directories under which to store log files

log.dirs=/tmp/kafka-logs-1

server2.properties

......

# The id of the broker. This must be set to a unique integer for each broker.

broker.id=2

......

# listeners = PLAINTEXT://your.host.name:9092

#listeners=PLAINTEXT://:9092

listeners=PLAINTEXT://:9094

......

# A comma seperated list of directories under which to store log files

log.dirs=/tmp/kafka-logs-2

(2)启动 Kafka broker

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server1.properties

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-server-start.sh config/server2.properties

(3)新建Kafka Topic

新建Topic Kafka-topic1:

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-topics.sh --create --zookeeper

localhost:2181 --replication-factor 2 --partitions 1 --topic kafka-topic1

Created topic "kafka-topic1".

检查Topic:

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-topics.sh --describe --zookeeper

localhost:2181

Topic:kafka-topic PartitionCount:1 ReplicationFactor:1 Configs:

Topic: kafka-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0

Topic:kafka-topic1 PartitionCount:1 ReplicationFactor:2 Configs:

Topic: kafka-topic1 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2

Topic:topic-name PartitionCount:1 ReplicationFactor:1 Configs:

Topic: topic-name Partition: 0 Leader: 0 Replicas: 0 Isr: 0

(4)启动Producer

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list

localhost:9093--topic kafka-topic1

yjc@linux-42ti:/install/kafka_2.12-1.0.0>bin/kafka-console-producer.sh --broker-list

localhost:9094--topic kafka-topic1

(5)启动Consumer

yjc@linux-42ti:/install/kafka_2.12-1.0.0> bin/kafka-console-consumer.sh --zookeeper

localhost:2181 --topic kafka-topic1 --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

(6)消息测试

END

  • 发表于:
  • 原文链接:https://kuaibao.qq.com/s/20180705G1404H00?refer=cp_1026
  • 腾讯「云+社区」是腾讯内容开放平台帐号(企鹅号)传播渠道之一,根据《腾讯内容开放平台服务协议》转载发布内容。

扫码关注云+社区

领取腾讯云代金券