ZooKeeper在线迁移实验
原创
注意事项
首先，当我们要从3台扩充到5台时，应保证集群不停止服务。
3台不停止服务的最低限度是2台（X/2+1），而5台的最低限度是3台。
- 我们应该保证，集群中最低有3台ZooKeeper是启动的。
- 此外，重启时应保证先重启
myid最小的机器，由小向大进行重启 -
Leader无论其myid大小，都放到最后重启
因为ZooKeeper的机制中，myid大的会向小的发起连接，而小的不会向大的发起连接。因此如果最后重启myid最小的机器，则其可能无法加入集群
环境情况
五台机器
IP | Hostname |
|---|---|
10.1.24.110 | idc02-kafka-ds-00 |
10.1.24.111 | idc02-kafka-ds-01 |
10.1.24.112 | idc02-kafka-ds-02 |
10.1.24.113 | idc02-kafka-ds-03 |
10.1.24.114 | idc02-kafka-ds-04 |
JDK
jdk1.7.0_67
ZooKeeper
zookeeper-3.4.6
myid
根据IP自增为1-5
配置文件
1 2 3 4 | server.1=10.1.24.110:2888:3888 server.2=10.1.24.111:2888:3888 server.3=10.1.24.112:2888:3888 |
|---|
实验过程
配置一个3节点的ZooKeeper
idc02-kafka-ds-00：
1 2 3 4 5 | [hadoop@idc02-kafka-ds-00 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
idc02-kafka-ds-01：
1 2 3 4 5 | [hadoop@idc02-kafka-ds-01 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: leader |
|---|
idc02-kafka-ds-02：
1 2 3 4 5 | [hadoop@idc02-kafka-ds-02 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
将其扩容为5节点的ZooKeeper
先查看原先的ZooKeeper集群情况
echo mntr|nc localhost 2181
这条4字命令可以查看集群的情况，其中follower的相关数据需要在Leader机器上才能查看
在idc02-kafka-ds-01上查看
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | [hadoop@idc02-kafka-ds-01 bin]$ echo mntr|nc localhost 2181 zk_version 3.4.6-1569965, built on 02/20/2014 09:09 GMT zk_avg_latency 0 zk_max_latency 0 zk_min_latency 0 zk_packets_received 3 zk_packets_sent 2 zk_num_alive_connections 1 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 4 zk_watch_count 0 zk_ephemerals_count 0 zk_approximate_data_size 27 zk_open_file_descriptor_count 27 zk_max_file_descriptor_count 65535 zk_followers 2 zk_synced_followers 2 zk_pending_syncs 0 |
|---|
启动另外两台机器的Zookeeper
另外两台机器的配置文件
1 2 3 4 5 6 | server.1=10.1.24.110:2888:3888 server.2=10.1.24.111:2888:3888 server.3=10.1.24.112:2888:3888 server.4=10.1.24.113:2888:3888 server.5=10.1.24.114:2888:3888 |
|---|
启动
idc02-kafka-ds-03：
1 2 3 4 5 | [hadoop@idc02-kafka-ds-03 bin]# ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
idc02-kafka-ds-04：
1 2 3 4 5 | [hadoop@idc02-kafka-ds-04 bin]# ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
再查看集群情况
仍然在idc02-kafka-ds-01上查看
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | [hadoop@idc02-kafka-ds-01 bin]$ echo mntr|nc localhost 2181 zk_version 3.4.6-1569965, built on 02/20/2014 09:09 GMT zk_avg_latency 0 zk_max_latency 0 zk_min_latency 0 zk_packets_received 4 zk_packets_sent 3 zk_num_alive_connections 1 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 4 zk_watch_count 0 zk_ephemerals_count 0 zk_approximate_data_size 27 zk_open_file_descriptor_count 31 zk_max_file_descriptor_count 65535 zk_followers 4 zk_synced_followers 4 zk_pending_syncs 0 |
|---|
可以看到zk_followers为4，连接到的follower从2变为4了
而且zk_synced_followers为4，说明新加入的2个也都同步好了
接下来我们滚动重启myid为1-3的前三台机器
先处理idc02-kafka-ds-00
关闭
如不放心请在关闭其间于
Leader机器或后加入的两台机器上监控日志
1 2 3 4 5 | [hadoop@idc02-kafka-ds-00 bin]$ ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
修改其配置文件
由原来的
1 2 3 4 | server.1=10.1.24.110:2888:3888 server.2=10.1.24.111:2888:3888 server.3=10.1.24.112:2888:3888 |
|---|
到新的
1 2 3 4 5 6 | server.1=10.1.24.110:2888:3888 server.2=10.1.24.111:2888:3888 server.3=10.1.24.112:2888:3888 server.4=10.1.24.113:2888:3888 server.5=10.1.24.114:2888:3888 |
|---|
启动
1 2 3 4 5 6 7 8 9 | [hadoop@idc02-kafka-ds-00 bin]$ ./zkServer.sh start JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [hadoop@idc02-kafka-ds-00 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
然后跳过作为Leader的idc02-kafka-ds-01，先处理idc02-kafka-ds-02
关闭
1 2 3 4 5 | [hadoop@idc02-kafka-ds-02 bin]$ ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
修改配置文件
1 2 3 4 5 6 | server.1=10.1.24.110:2888:3888 server.2=10.1.24.111:2888:3888 server.3=10.1.24.112:2888:3888 server.4=10.1.24.113:2888:3888 server.5=10.1.24.114:2888:3888 |
|---|
启动
1 2 3 4 5 6 7 8 9 | [hadoop@idc02-kafka-ds-02 bin]$ ./zkServer.sh start JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [hadoop@idc02-kafka-ds-02 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
最后处理原Leader的idc02-kafka-ds-01
关闭
1 2 3 4 5 | [hadoop@idc02-kafka-ds-01 bin]$ ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
查看新Leader
ZooKeeper会尽可能的选择myid最大的机器为Leader，因此原本的idc02-kafka-ds-04其myid为5变为了Leader
1 2 3 4 5 | [hadoop@idc02-kafka-ds-04 bin]# ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: leader |
|---|
修改配置文件
1 2 3 4 5 6 | server.1=10.1.24.110:2888:3888 server.2=10.1.24.111:2888:3888 server.3=10.1.24.112:2888:3888 server.4=10.1.24.113:2888:3888 server.5=10.1.24.114:2888:3888 |
|---|
启动
1 2 3 4 5 6 7 8 9 | [hadoop@idc02-kafka-ds-01 bin]$ ./zkServer.sh start JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [hadoop@idc02-kafka-ds-01 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
在新的Leader上查看集群情况
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | [hadoop@idc02-kafka-ds-04 bin]# echo mntr|nc localhost 2181 zk_version 3.4.6-1569965, built on 02/20/2014 09:09 GMT zk_avg_latency 1 zk_max_latency 4 zk_min_latency 0 zk_packets_received 12 zk_packets_sent 11 zk_num_alive_connections 1 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 4 zk_watch_count 0 zk_ephemerals_count 0 zk_approximate_data_size 27 zk_open_file_descriptor_count 33 zk_max_file_descriptor_count 65535 zk_followers 4 zk_synced_followers 4 zk_pending_syncs 0 |
|---|
一切正常 到这里，我们已经将原本的3台扩展到了5台，成功了一半。 然后只要将现在的5台再缩小到3台且不包括原本
myid为1-2的机器，就完成了迁移
将5台缩小回3台
修改idc02-kafka-ds-02
根据前面的注意事项，我们此时5台集群中启动的数量不得少于3台，因此我们需要先修改
3-5号机器的配置文件为3台，再关闭1-2号机器
关闭
1 2 3 4 5 | [hadoop@idc02-kafka-ds-02 bin]$ ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
修改配置文件为
1 2 3 4 | server.3=10.1.24.110:2888:3888 server.4=10.1.24.111:2888:3888 server.5=10.1.24.112:2888:3888 |
|---|
启动
l
1 2 3 4 5 6 7 8 9 | [hadoop@idc02-kafka-ds-02 bin]$ ./zkServer.sh start JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [hadoop@idc02-kafka-ds-02 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
然后修改idc02-kafka-ds-03
关闭
1 2 3 4 5 | [hadoop@idc02-kafka-ds-03 bin]# ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
修改配置文件为
1 2 3 4 | server.3=10.1.24.110:2888:3888 server.4=10.1.24.111:2888:3888 server.5=10.1.24.112:2888:3888 |
|---|
启动
1 2 3 4 5 6 7 8 9 | [hadoop@idc02-kafka-ds-03 bin]$ ./zkServer.sh start JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [hadoop@idc02-kafka-ds-03 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
最后修改idc02-kafka-ds-04
关闭
1 2 3 4 5 | [hadoop@idc02-kafka-ds-04 bin]$ ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
关闭后
Leader移动到了myid第二大的idc02-kafka-ds-02上
修改配置文件为
l
1 2 3 4 | server.3=10.1.24.110:2888:3888 server.4=10.1.24.111:2888:3888 server.5=10.1.24.112:2888:3888 |
|---|
启动
1 2 3 4 5 6 7 8 9 | [hadoop@idc02-kafka-ds-04 bin]$ ./zkServer.sh start JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [hadoop@idc02-kafka-ds-04 bin]$ ./zkServer.sh status JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower |
|---|
在Leader中查看
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | [hadoop@idc02-kafka-ds-03 bin]$ echo mntr|nc localhost 2181 zk_version 3.4.6-1569965, built on 02/20/2014 09:09 GMT zk_avg_latency 0 zk_max_latency 0 zk_min_latency 0 zk_packets_received 4 zk_packets_sent 3 zk_num_alive_connections 1 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 4 zk_watch_count 0 zk_ephemerals_count 0 zk_approximate_data_size 27 zk_open_file_descriptor_count 27 zk_max_file_descriptor_count 65535 zk_followers 2 zk_synced_followers 2 zk_pending_syncs 0 |
|---|
此时的zk_followers为2，说明Leader已经不认1-2号机器了
关闭1-2号机器
关闭idc02-kafka-ds-00
1 2 3 4 5 | [hadoop@idc02-kafka-ds-00 bin]$ ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
关闭idc02-kafka-ds-01
1 2 3 4 5 | [hadoop@idc02-kafka-ds-01 bin]$ ./zkServer.sh stop JMX enabled by default Using config: /usr/local/webserver/zookeeper-3.4.6/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED |
|---|
再查看
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | [hadoop@idc02-kafka-ds-03 bin]$ echo mntr|nc localhost 2181 zk_version 3.4.6-1569965, built on 02/20/2014 09:09 GMT zk_avg_latency 0 zk_max_latency 0 zk_min_latency 0 zk_packets_received 5 zk_packets_sent 4 zk_num_alive_connections 1 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 4 zk_watch_count 0 zk_ephemerals_count 0 zk_approximate_data_size 27 zk_open_file_descriptor_count 27 zk_max_file_descriptor_count 65535 zk_followers 2 zk_synced_followers 2 zk_pending_syncs 0 |
|---|
没有任何影响 实验成功
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。
如有侵权，请联系 cloudcommunity@tencent.com 删除。
腾讯云开发者
