搜索
创作
提问
登录注册
展开

专栏首页瓜农老梁Kafka集群平滑扩容及Leader均衡【实战笔记】

Kafka集群平滑扩容及Leader均衡【实战笔记】

阅读 5800
一、问题描述

当集群中新增加节点时，需要对已有的topic的副本进行迁移，以平衡流量。以公司集群扩增两个节点broker 4和broker 5为例说明操作过程。

问题：怎么做才能做到平滑呢？即尽量做到客户端应用无感知。

为了解决平滑问题，分为三步完成

  • 1.副本均衡设置 对Topic的副本平均分配到各个broker上
  • 2.偏好副本设置 将偏好副本平均分配到各个broker上, 为Leader均衡做准备
  • 3.Leader均衡 执行Leader平衡

要点备注

  • 第一步副本均衡操作要点 为了不影响客户端使用，保持原有集群Leader副本现状不变，将其他副本平均分配到各个broker上。 副本均衡设置后，需要等待与观察，最终让这些设置的副本进入ISR列表后（新加入的副本跟上了Leader副本数据进度）再执行第二步。
  • 第二步偏好副本操作要点 在第二步偏好副本设置时，将偏好副本均匀的分布broker上，每个broker上的偏好副本数量=分区总数/broker数量
  • 第三步Leader均衡操作要点 这一步开始实际leader均衡操作
二、副本均衡设置

1.查看当前Topic的副本分配情况

bin/kafka-topics.sh --describe --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --topic zto_sign_disfee

Topic:zto_sign_disfee   PartitionCount:12       ReplicationFactor:3     Configs:
        Topic: zto_sign_disfee  Partition: 0    Leader: 2       Replicas: 2,0,3 Isr: 3,0,2
        Topic: zto_sign_disfee  Partition: 1    Leader: 0       Replicas: 0,1,3 Isr: 3,0,1
        Topic: zto_sign_disfee  Partition: 2    Leader: 1       Replicas: 1,2,3 Isr: 3,1,2
        Topic: zto_sign_disfee  Partition: 3    Leader: 2       Replicas: 2,1,3 Isr: 3,1,2
        Topic: zto_sign_disfee  Partition: 4    Leader: 0       Replicas: 0,2,3 Isr: 3,0,2
        Topic: zto_sign_disfee  Partition: 5    Leader: 1       Replicas: 1,0,2 Isr: 0,1,2
        Topic: zto_sign_disfee  Partition: 6    Leader: 2       Replicas: 2,0,1 Isr: 0,1,2
        Topic: zto_sign_disfee  Partition: 7    Leader: 0       Replicas: 0,1,3 Isr: 3,0,1
        Topic: zto_sign_disfee  Partition: 8    Leader: 1       Replicas: 1,2,0 Isr: 0,1,2
        Topic: zto_sign_disfee  Partition: 9    Leader: 2       Replicas: 2,1,3 Isr: 3,1,2
        Topic: zto_sign_disfee  Partition: 10   Leader: 0       Replicas: 0,2,3 Isr: 3,0,2
        Topic: zto_sign_disfee  Partition: 11   Leader: 1       Replicas: 1,0,3 Isr: 3,0,1

2.准备执行计划的Topic

echo '{"version":1,"topics":[{"topic":"zto_sign_disfee"}]}' > plan02/zto_sign_disfee.json

3.生成执行计划

bin/kafka-reassign-partitions.sh --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --topics-to-move-json-file plan02/zto_sign_disfee.json --broker-list "0,1,2,3,4,5" --generate

Current partition replica assignment
{"version":1,"partitions":[{"topic":"zto_sign_disfee","partition":11,"replicas":[1,0,3]},{"topic":"zto_sign_disfee","partition":7,"replicas":[0,1,3]},{"topic":"zto_sign_disfee","partition":3,"replicas":[2,1,3]},{"topic":"zto_sign_disfee","partition":1,"replicas":[0,1,3]},{"topic":"zto_sign_disfee","partition":2,"replicas":[1,2,3]},{"topic":"zto_sign_disfee","partition":5,"replicas":[1,0,2]},{"topic":"zto_sign_disfee","partition":4,"replicas":[0,2,3]},{"topic":"zto_sign_disfee","partition":0,"replicas":[2,0,3]},{"topic":"zto_sign_disfee","partition":9,"replicas":[2,1,3]},{"topic":"zto_sign_disfee","partition":8,"replicas":[1,2,0]},{"topic":"zto_sign_disfee","partition":6,"replicas":[2,0,1]},{"topic":"zto_sign_disfee","partition":10,"replicas":[0,2,3]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"zto_sign_disfee","partition":11,"replicas":[4,3,5]},{"topic":"zto_sign_disfee","partition":7,"replicas":[0,5,1]},{"topic":"zto_sign_disfee","partition":1,"replicas":[0,4,5]},{"topic":"zto_sign_disfee","partition":3,"replicas":[2,0,1]},{"topic":"zto_sign_disfee","partition":4,"replicas":[3,1,2]},{"topic":"zto_sign_disfee","partition":2,"replicas":[1,5,0]},{"topic":"zto_sign_disfee","partition":5,"replicas":[4,2,3]},{"topic":"zto_sign_disfee","partition":0,"replicas":[5,3,4]},{"topic":"zto_sign_disfee","partition":9,"replicas":[2,1,3]},{"topic":"zto_sign_disfee","partition":8,"replicas":[1,0,2]},{"topic":"zto_sign_disfee","partition":6,"replicas":[5,4,0]},{"topic":"zto_sign_disfee","partition":10,"replicas":[3,2,4]}]}

4.重新分配副本

zto_sign_disfee-reassign.json

{
    "version": 1,
    "partitions": [{
        "topic": "zto_sign_disfee",
        "partition": 11,
        "replicas": [1,
        4,
        3]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 7,
        "replicas": [0,
        4,
        3]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 3,
        "replicas": [2,
        4,
        3]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 1,
        "replicas": [0,
        4,
        3]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 2,
        "replicas": [1,
        5,
        3]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 5,
        "replicas": [1,
        4,
        2]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 4,
        "replicas": [0,
        5,
        3]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 0,
        "replicas": [2,
        4,
        5]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 9,
        "replicas": [2,
        1,
        5]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 8,
        "replicas": [1,
        5,
        0]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 6,
        "replicas": [2,
        0,
        1]
    },
    {
        "topic": "zto_sign_disfee",
        "partition": 10,
        "replicas": [0,
        2,
        5]
    }]
}

5.执行计划

bin/kafka-reassign-partitions.sh --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --reassignment-json-file plan02/zto_sign_disfee-reassign.json --execute

Current partition replica assignment
{"version":1,"partitions":[{"topic":"zto_sign_disfee","partition":11,"replicas":[1,0,3]},{"topic":"zto_sign_disfee","partition":7,"replicas":[0,1,3]},{"topic":"zto_sign_disfee","partition":3,"replicas":[2,1,3]},{"topic":"zto_sign_disfee","partition":1,"replicas":[0,1,3]},{"topic":"zto_sign_disfee","partition":2,"replicas":[1,2,3]},{"topic":"zto_sign_disfee","partition":5,"replicas":[1,0,2]},{"topic":"zto_sign_disfee","partition":4,"replicas":[0,2,3]},{"topic":"zto_sign_disfee","partition":0,"replicas":[2,0,3]},{"topic":"zto_sign_disfee","partition":9,"replicas":[2,1,3]},{"topic":"zto_sign_disfee","partition":8,"replicas":[1,2,0]},{"topic":"zto_sign_disfee","partition":6,"replicas":[2,0,1]},{"topic":"zto_sign_disfee","partition":10,"replicas":[0,2,3]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

6.验证

bin/kafka-reassign-partitions.sh --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --reassignment-json-file plan02/zto_sign_disfee-reassign.json --verify

Status of partition reassignment:
Reassignment of partition [zto_sign_disfee,9] completed successfully
Reassignment of partition [zto_sign_disfee,10] completed successfully
Reassignment of partition [zto_sign_disfee,6] completed successfully
Reassignment of partition [zto_sign_disfee,7] completed successfully
Reassignment of partition [zto_sign_disfee,5] completed successfully
Reassignment of partition [zto_sign_disfee,3] completed successfully
Reassignment of partition [zto_sign_disfee,11] completed successfully
Reassignment of partition [zto_sign_disfee,1] completed successfully
Reassignment of partition [zto_sign_disfee,8] completed successfully
Reassignment of partition [zto_sign_disfee,2] completed successfully
Reassignment of partition [zto_sign_disfee,0] completed successfully
Reassignment of partition [zto_sign_disfee,4] completed successfully

7.查看topic副本情况

bin/kafka-topics.sh --describe --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --topic zto_sign_disfee

Topic:zto_sign_disfee   PartitionCount:12       ReplicationFactor:3     Configs:
        Topic: zto_sign_disfee  Partition: 0    Leader: 2       Replicas: 2,4,5 Isr: 5,2,4
        Topic: zto_sign_disfee  Partition: 1    Leader: 0       Replicas: 0,4,3 Isr: 3,0,4
        Topic: zto_sign_disfee  Partition: 2    Leader: 1       Replicas: 1,5,3 Isr: 3,1,5
        Topic: zto_sign_disfee  Partition: 3    Leader: 2       Replicas: 2,4,3 Isr: 3,2,4
        Topic: zto_sign_disfee  Partition: 4    Leader: 0       Replicas: 0,5,3 Isr: 3,0,5
        Topic: zto_sign_disfee  Partition: 5    Leader: 1       Replicas: 1,4,2 Isr: 1,2,4
        Topic: zto_sign_disfee  Partition: 6    Leader: 2       Replicas: 2,0,1 Isr: 0,1,2
        Topic: zto_sign_disfee  Partition: 7    Leader: 0       Replicas: 0,4,3 Isr: 3,0,4
        Topic: zto_sign_disfee  Partition: 8    Leader: 1       Replicas: 1,5,0 Isr: 0,1,5
        Topic: zto_sign_disfee  Partition: 9    Leader: 2       Replicas: 2,1,5 Isr: 1,2,5
        Topic: zto_sign_disfee  Partition: 10   Leader: 0       Replicas: 0,2,5 Isr: 0,2,5
        Topic: zto_sign_disfee  Partition: 11   Leader: 1       Replicas: 1,4,3 Isr: 3,1,4
三、偏好副本设置

1.准备计划文件

zto_sign_disfee-perf-reassign.json

{"version":1,"partitions":[{"topic":"zto_sign_disfee","partition":11,"replicas":[3,4,1]},{"topic":"zto_sign_disfee","partition":7,"replicas":[4,0,3]},{"topic":"zto_sign_disfee","partition":3,"replicas":[3,4,2]},{"topic":"zto_sign_disfee","partition":1,"replicas":[4,0,3]},{"topic":"zto_sign_disfee","partition":2,"replicas":[5,1,3]},{"topic":"zto_sign_disfee","partition":5,"replicas":[1,4,2]},{"topic":"zto_sign_disfee","partition":4,"replicas":[0,5,3]},{"topic":"zto_sign_disfee","partition":0,"replicas":[2,4,5]},{"topic":"zto_sign_disfee","partition":9,"replicas":[2,1,5]},{"topic":"zto_sign_disfee","partition":8,"replicas":[5,1,0]},{"topic":"zto_sign_disfee","partition":6,"replicas":[1,0,2]},{"topic":"zto_sign_disfee","partition":10,"replicas":[0,2,5]}]}

2.执行与验证

bin/kafka-reassign-partitions.sh --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --reassignment-json-file plan02/zto_sign_disfee-perf-reassign.json --execute

bin/kafka-reassign-partitions.sh --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --reassignment-json-file plan02/zto_sign_disfee-perf-reassign.json --verify

四、Leader均衡

1.准备计划文件

zto_sign_disfee-leader-reassign.json

{"partitions":[{"topic":"zto_sign_disfee","partition":0},{"topic":"zto_sign_disfee","partition":1},{"topic":"zto_sign_disfee","partition":2},{"topic":"zto_sign_disfee","partition":3},{"topic":"zto_sign_disfee","partition":4},{"topic":"zto_sign_disfee","partition":5},{"topic":"zto_sign_disfee","partition":6},{"topic":"zto_sign_disfee","partition":7},{"topic":"zto_sign_disfee","partition":8},{"topic":"zto_sign_disfee","partition":9},{"topic":"zto_sign_disfee","partition":10},{"topic":"zto_sign_disfee","partition":11}]}

2.执行计划

bin/kafka-preferred-replica-election.sh --zookeeper 192.168.x.x:2181,192.168.x.x:2181,192.168.x.x:2181 --path-to-json-file plan02/zto_sign_disfee-leader-reassign.json

本文分享自微信公众号 - 瓜农老梁（gh_01130ae30a83），作者：梁勇

原文出处及转载信息见文内详细说明，如有侵权，请联系 yunjia_community@tencent.com 删除。

原始发表时间：2019-08-30

本文参与腾讯云自媒体分享计划，欢迎正在阅读的你也加入，一起分享。

展开阅读全文
举报

我来说两句

0 条评论
登录 后参与评论

相关文章

  • RocketMQ消息发送【源码笔记】（一）

    b.MQClientInstance封装了网络通信的管道，存储于factoryTable（ConcurrentHashMap）c.factoryTable为MQ...

    瓜农老梁

  • RocketMQ消息发送常见错误与解决方案

    本文将结合自己使用RocketMQ的经验，对消息发送常见的问题进行分享，基本会遵循出现问题，分析问题、解决问题。

    瓜农老梁

  • Java NIO通道概览与文件通道【源码笔记】

    系统I/O即字节的传输，Channel即传输的通道，文件或网络Socket服务即传输的目的地。

    瓜农老梁

  • 一句SQL完成动态分级查询

    在最近的活字格项目中使用ActiveReports报表设计器设计一个报表模板时，遇到一个多级分类的难题：需要将某个部门所有销售及下属部门的销售金额汇总，因为下属...

    葡萄城控件

  • day16_ajax学习笔记

    JavaScript 中 XMLHttpRequest对象是整个Ajax技术的核心，它提供了异步发送请求的能力。

    黑泽君

  • 【AI版狄仁杰】爱丁堡大学团队教AI识别犯罪剧中的罪犯，正确率达60%仍不及人类

    作者：常佩琦 【新智元导读】爱丁堡大学的研究人员通过《犯罪现场调查》系列电视剧的相关视频、音频和文本训练AI，测试AI能否准确识别每集中的罪犯。在每集电视剧结束...

    新智元

  • PHP中Smarty引擎的常用语法

    后天Date of the day after tomorrow (Day+2):

    Enjoy233

  • 【pygame系列 第三课 弹球游戏-上 】

    python我们可以做文字版的游戏，比如猜数字游戏，21点游戏。那python可以做图形界面的游戏吗？偷偷告诉你，用pygame库就可以实现了。pygame是p...

    用户7054460

  • R 热图绘制heatmap②

    用户1359560

  • 生信代码：“热图”来袭（pheatmap）

    热图可以聚合大量的数据，并可以用一种渐进色来优雅地表现，可以很直观地展现数据的疏密程度或频率高低。

    科研菌
更多文章

作者介绍

瓜农老梁

专栏

精选专题

活动推荐

运营活动

活动名称
广告关闭

扫码关注云+社区

领取腾讯云代金券