kafka增加topic的备份数量

一、困难点

建立topic的时候,可以通过指定参数 --replication-factor 设置备份数量。但是,一旦完成建立topic,则无法通过kafka-topic.sh 或者 命令修改replica数量。

二、解决办法

实际上,我们可以考虑一种 “另类” 的办法:可以利用 kafka-reassign-partitions.sh 命令对所有分区进行重新分布,在做分区重新分布的时候,通过增加每个分区的replica备份数量来达到目的。

本文将介绍如何利用 kafka-reassign-partitions.sh 命令增加topic的备份数量。

注意:以下命令使用到的topic名称、zookeeper的ip和port,需要读者替换成为实际集群的参数。

(假设kafka集群有4个broker,id分别为:1001,1002,1003,1004)

2.1、获取当前topic的所有分区分布在broker的情况

[root@tbds bin]#./kafka-topics.sh--zookeeper172.16.32.13:2181--topic ranger_audits--describeTopic:ranger_audits PartitionCount:10ReplicationFactor:1Configs:Topic:ranger_audits Partition:Leader:1001Replicas:1001Isr:1001Topic:ranger_audits Partition:1Leader:1002Replicas:1002Isr:1002Topic:ranger_audits Partition:2Leader:1001Replicas:1001Isr:1001Topic:ranger_audits Partition:3Leader:1002Replicas:1002Isr:1002Topic:ranger_audits Partition:4Leader:1001Replicas:1001Isr:1001Topic:ranger_audits Partition:5Leader:1002Replicas:1002Isr:1002Topic:ranger_audits Partition:6Leader:1001Replicas:1001Isr:1001Topic:ranger_audits Partition:7Leader:1002Replicas:1002Isr:1002Topic:ranger_audits Partition:8Leader:1001Replicas:1001Isr:1001Topic:ranger_audits Partition:9Leader:1002Replicas:1002Isr:1002

可以看出,ranger_audits 这个topic有10个分区,每个分区只有一个feplica备份,分布在1001和1002两台broker上面。

下面我们需要将ranger_audits 的每个分区数据都增加到2个replica备份,且分布到4个broker上面。

2.2、创建增加replica备份数量的配置文件

(注意:尽量保持topic的原有每个分区的主备份不变化。因此,配置文件的每个分区的第一个broker保持不变。)

[root@tbds bin]# vim../config/increase-replication-factor.json{"version":1,"partitions":[{"topic":"ranger_audits","partition":,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}]}

上面的配置文件说明,我们将topic的每个分区都增加了一个replica,且保持每个分区原有的主备份所在broker不变化,将每个分区新增的replica备份数据放到到1003和1004两个broker上面。

2.3、开始执行增加分区

[root@tbds bin]# ./kafka-reassign-partitions.sh-zookeeper172.16.32.13:2181--reassignment-json-file ../config/increase-replication-factor.json--executeCurrent partition replica assignment{"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}Save thistouseasthe--reassignment-json-fileoptionduring rollbackSuccessfully started reassignmentofpartitions {"version":1,"partitions":[{"topic":"ranger_audits","partition":,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}

2.4、查看执行进度

[root@tbds bin]#./kafka-reassign-partitions.sh-zookeeper172.16.32.13:2181--reassignment-json-file../config/increase-replication-factor.json--verifyStatusofpartition reassignment:Reassignmentofpartition[ranger_audits,]completed successfullyReassignmentofpartition[ranger_audits,8]completed successfullyReassignmentofpartition[ranger_audits,5]completed successfullyReassignmentofpartition[ranger_audits,2]completed successfullyReassignmentofpartition[ranger_audits,9]completed successfullyReassignmentofpartition[ranger_audits,1]completed successfullyReassignmentofpartition[ranger_audits,3]completed successfullyReassignmentofpartition[ranger_audits,4]completed successfullyReassignmentofpartition[ranger_audits,7]completed successfullyReassignmentofpartition[ranger_audits,6]completed successfully

上面显示增加分区操作成功

2.5、再次查看topic的情况

[root@tbds bin]#./kafka-topics.sh--zookeeper172.16.32.13:2181--topic ranger_audits--describeTopic:ranger_audits PartitionCount:10ReplicationFactor:2Configs:Topic:ranger_audits Partition:Leader:1001Replicas:1001,1003Isr:1001,1003Topic:ranger_audits Partition:1Leader:1002Replicas:1002,1004Isr:1002,1004Topic:ranger_audits Partition:2Leader:1001Replicas:1001,1003Isr:1001,1003Topic:ranger_audits Partition:3Leader:1002Replicas:1002,1004Isr:1002,1004Topic:ranger_audits Partition:4Leader:1001Replicas:1001,1003Isr:1001,1003Topic:ranger_audits Partition:5Leader:1002Replicas:1002,1004Isr:1002,1004Topic:ranger_audits Partition:6Leader:1001Replicas:1001,1003Isr:1001,1003Topic:ranger_audits Partition:7Leader:1002Replicas:1002,1004Isr:1002,1004Topic:ranger_audits Partition:8Leader:1001Replicas:1001,1003Isr:1001,1003Topic:ranger_audits Partition:9Leader:1002Replicas:1002,1004Isr:1002,1004

从上面可以看出,备份数量增加成功

三、进一步思考

利用上述介绍的办法,除了可以用来增加topic的备份数量之外,还能够处理以下几个场景:

1、对topic的所有分区数据进行整体迁移。怎么理解呢?假如集群有N个broker,后来新扩容M个broker。由于新扩容的broker磁盘都是空的,原有的broker磁盘占用都很满。那么我们可以利用上述方法,将存储在原有N个broker的某些topic整体搬迁到新扩容的M个broker,进而实现kafka集群的整体数据均衡。

具体使用方法就是:通过编写2.2章节的配置文件,将topic的所有分区都配置到新的M个broker上面去,再执行excute,即可完成topic的所有分区数据整体迁移到新扩容的M个broker节点。

2、broker坏掉的情况。导致某些topic的某些分区的replica数量减少,可以利用kafka-reassign-partitions.sh增加replica;

3、kafka 某些broker磁盘占用很满,某些磁盘占用又很少。可以利用kafka-reassign-partitions.sh迁移某些topic的分区数据到磁盘占用少的broker,实现数据均衡;

4、kafka集群扩容。需要把原来broker的topic数据整体迁移到新的broker,合理利用新扩容的broker,实现负载均衡。

  • 发表于:
  • 原文链接https://kuaibao.qq.com/s/20190131A00BJ200?refer=cp_1026
  • 腾讯「云+社区」是腾讯内容开放平台帐号(企鹅号)传播渠道之一,根据《腾讯内容开放平台服务协议》转载发布内容。

同媒体快讯

扫码关注云+社区

领取腾讯云代金券