Kafka集群新增节点后数据如何重分配
新增节点的步骤
将其他节点的server.properties配置文件拷贝后修改以下参数
broker.id
log.dirs
zookeeper.connect
数据迁移原理
- 只有新增的Topic才会将数据分布在新节点上，如果要将现有数据也分配到新节点，需要将Topic中的数据迁移到新节点上。
- 数据迁移过程是手动启动的，但是是完全自动化的。Kafka会将新节点添加为要迁移的分区的追随者，并允许其完全复制该分区中的现有数据。新节点完全复制此分区的内容并加入同步副本后，现有副本之一将删除其分区的数据。
数据迁移工具介绍
分区重新分配工具可用于在代理之间移动分区。理想的分区分配将确保所有代理之间的数据负载和分区大小均匀。分区重新分配工具没有能力自动研究Kafka群集中的数据分布，并四处移动分区以实现均匀的负载分布。因此，必须弄清楚应该移动哪些主题或分区。
分区重新分配工具可以在3种模式下运行：
- --generate：在此模式下，给定主题列表和代理列表，该工具会生成分区与副本重新分配的计划，以将指定主题的所有分区在所有节点上重新分配。在给定主题和目标代理的列表的情况下，此选项仅提供了一种方便的方式来生成分区重新分配计划。
- --execute：在此模式下，该工具将根据用户提供的重新分配计划启动分区的重新分配。（使用--reassignment-json-file选项）。这可以是管理员手工制作的自定义重新分配计划，也可以使用--generate选项提供
- --verify：在此模式下，该工具会验证上一次--execute期间列出的所有分区的重新分配状态。状态可以是成功完成，失败或进行中
示例：
现有5个节点的broker_id为1,2,3,4,5；新增节点broker_id为6
Topic：test 有6个分区,5个副本
创建要迁移的topic配置文件
topics-to-move.json
{"topics": [{"topic": "test"}],"version":1}
生成重新分配计划
kafka-reassign-partitions --bootstrap-server localhost:9092 --zookeeper zookeeper-001:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5,6" --generate
以上命令将会产生以下内容，将 Proposed partition reassignment configuration 下的内容保存为test-reassign.json文件
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[5,4,1,2,3],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":5,"replicas":[5,2,3,4,1],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":1,"replicas":[1,5,2,3,4],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":4,"replicas":[4,5,1,2,3],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":3,"replicas":[3,4,5,1,2],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":2,"replicas":[2,1,3,4,5],"log_dirs":["any","any","any","any","any"]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test","partition":4,"replicas":[5,1,2,3,4],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":1,"replicas":[2,4,5,6,1],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":3,"replicas":[4,6,1,2,3],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":0,"replicas":[1,3,4,5,6],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":5,"replicas":[6,2,3,4,5],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":2,"replicas":[3,5,6,1,2],"log_dirs":["any","any","any","any","any"]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test","partition":4,"replicas":[5,1,2,3,4],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":1,"replicas":[2,4,5,6,1],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":3,"replicas":[4,6,1,2,3],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":0,"replicas":[1,3,4,5,6],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":5,"replicas":[6,2,3,4,5],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":2,"replicas":[3,5,6,1,2],"log_dirs":["any","any","any","any","any"]}]}
执行数据迁移
kafka-reassign-partitions --bootstrap-server localhost:9092 --zookeeper zookeeper-001:2181 --reassignment-json-file test-reassign.json --execute
检查重新分配的分区状态
kafka-reassign-partitions --bootstrap-server localhost:9092 --zookeeper zookeeper-001:2181 --reassignment-json-file test-reassign.json --verify本文参与 腾讯云自媒体同步曝光计划，分享自微信公众号。
原始发表：2022-02-14，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
目录
腾讯云开发者
