最近我们生产环境的Hadoop
集群需要调整几台服务器,具体转换关系如下:
datanode92.bi -> namenode02.bi
namenode01.bi(old) -> datanode19.bi
namenode02.bi -> datanode20.bi
最终目标为:
datanode92.bi
、namenode01.bi
服务器上的DataNode
服务下线namenode02.bi
服务器配置较低,因此将namenode02.bi
服务器上的NameNode
服务下线,将其迁移到配置更高的datanode92.bi
服务器上namenode01.bi(old)
服务器调整为DataNode
服务器datanode19.bi
、datanode20.bi
两个服务器名称1、在namenode01
上,添加退役节点的IP
到黑名单,/usr/local/hadoop-2.6.3/etc/hadoop/dfs.exclude
文件添加如下需要下线的服务器主机名称:
datanode92.bi
namenode01.bi
2、配置hdfs-site.xml
:
<property>
<name>dfs.hosts.exclude</name>
<value>/usr/local/hadoop-2.6.3/etc/hadoop/dfs.exclude</value>
</property>
3、切换到/usr/local/hadoop-2.6.3/bin
目录下,执行如下命令,刷新NN
节点,不需要重启NN
服务:
./hdfs dfsadmin -refreshNodes
执行命令后,需要及时查看NameNode
日志验证命令是否执行成功,hadoop-hadoop-namenode-namenode01.bi.log
输出日志如下:
Refresh nodes successful for namenode01.bi.10101111.com/10.216.2.25:8020
Refresh nodes successful for namenode02.bi.10101111.com/10.216.5.2:8020
4、查看spaceX
,观察状态在decommisstion in progress
的节点:
同时,还要查看需要进行拷贝的block
副本数,也就是Number of Under-Replicated Blocks
这个指标的值:
5、当所有要退役的节点状态都为Decommissioned
,且Number of Under-Replicated Blocks
数值为0
,表明数据迁移工作已经完成
6、从slaves
文件中清空退役节点
7、DN
下线完成后,spaceX
报磁盘满的警告,经过查看整个集群空间使用量及剩余量信息,证实可以忽略此警告:
(1)警告信息:
(2)磁盘剩余量17.46%
:
1、停止服务:
namenode02.bi
上的 namenode
、zkfc
、journalnode
、resourcemanager
服务,使namenode02.bi
上的修改记录数据和元数据不再更新:/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh stop namenode
/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh stop zkfc
/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh stop journalnode
Hive
服务:查出hiveserver2
和metastore
的进程号,直接kill
2、通知运维更新网络信息:
DNS
信息datanode92.bi
的主机名称修改为namenode02.bi
的主机名称,并同步更新Hadoop
集群服务器的host
文件、slaves
文件3、拷贝数据:
NameNode
的整个Hadoop
目录拷贝到目标机器的/usr/local
路径下/data/dfs
的元数据拷贝到目标机器的/data/dfs
路径下scp -r dfs hadoop@datanode92.bi:/data/dfs/
/usr/local/apache-hive-1.2.1-bin
拷贝到目标机器的/usr/local
路径下4、启动服务:
(1)检查配置文件,在目标机器上启动namenode
、zkfc
、journalnode
服务。以下是NameNode
服务启动后的块汇报日志,可以看到大量带有BlockStateChange
关键字的日志,这样的日志打印没完成,就证明NameNode
没有完全正常,不能对外提供服务,日志如下:
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.26:50010 is added to blk_2880883080_1896501829{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2297011b-a18a-449b-b795-248b44d082eb:NORMAL:10.216.5.26:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.21:50010 is added to blk_2880883080_1896501829{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2297011b-a18a-449b-b795-248b44d082eb:NORMAL:10.216.5.26:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.40:50010 is added to blk_2880883080_1896501829{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2297011b-a18a-449b-b795-248b44d082eb:NORMAL:10.216.5.26:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-0e5e6e27-8744-4e6a-8006-690f6e099676:NORMAL:10.216.5.40:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.21:50010 is added to blk_2880883081_1896501830{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.4:50010 is added to blk_2880883081_1896501830{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-fa7fa32c-91bb-47a3-a559-2fc31f74cdec:NORMAL:10.216.4.4:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.48:50010 is added to blk_2880883081_1896501830{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-fa7fa32c-91bb-47a3-a559-2fc31f74cdec:NORMAL:10.216.4.4:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-df7f2d4a-0b76-46ee-9e30-d42921ba825a:NORMAL:10.216.5.48:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.21:50010 is added to blk_2880883082_1896501831{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.19:50010 is added to blk_2880883082_1896501831{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-2c37fdd9-9d89-493c-bb0d-b915b9fd0476:NORMAL:10.216.4.19:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.41:50010 is added to blk_2880883082_1896501831{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-2c37fdd9-9d89-493c-bb0d-b915b9fd0476:NORMAL:10.216.4.19:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-91a83f26-10a5-4456-94ec-460d5a428990:NORMAL:10.216.4.41:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,796 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.34:50010 is added to blk_2880883083_1896501832{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-5540b823-7606-432a-98c9-a8d84c2c3e72:NORMAL:10.216.2.34:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,796 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.8:50010 is added to blk_2880883083_1896501832{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-5540b823-7606-432a-98c9-a8d84c2c3e72:NORMAL:10.216.2.34:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-110ef02a-b044-41bf-9d65-f5fdaa76b959:NORMAL:10.216.2.8:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,796 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.8:50010 is added to blk_2880883083_1896501832{blockUCState=UNDER_CONSTRUCTION, prima849 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883130_1896501879{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,863 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log: 132651/150565 transactions completed. (88%)
2019-03-12 16:20:43,879 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.38:50010 is added to blk_2880883136_1896501885{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9dec2025-66f8-4b4a-a197-e56751b284d5:NORMAL:10.216.4.38:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,893 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.5:50010 is added to blk_2880883140_1896501889{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,893 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.16:50010 is added to blk_2880883140_1896501889{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-32c95765-9f02-4863-8ba2-f1fecf2e0872:NORMAL:10.216.5.16:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,893 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.20:50010 is added to blk_2880883140_1896501889{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-32c95765-9f02-4863-8ba2-f1fecf2e0872:NORMAL:10.216.5.16:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-32001683-8bf1-4ae1-baf4-0efd7d117c94:NORMAL:10.216.2.20:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,904 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.38:50010 is added to blk_2880883143_1896501892{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9dec2025-66f8-4b4a-a197-e56751b284d5:NORMAL:10.216.4.38:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,905 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.23:50010 is added to blk_2880883144_1896501893{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68b7e245-3c92-4f81-95fe-dcb87e2453a8:NORMAL:10.216.2.23:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,905 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.22:50010 is added to blk_2880883144_1896501893{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68b7e245-3c92-4f81-95fe-dcb87e2453a8:NORMAL:10.216.2.23:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-6b7b44fb-539f-4a1c-a867-1a0d7fe77972:NORMAL:10.216.5.22:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,912 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883145_1896501894{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,912 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.22:50010 is added to blk_2880883145_1896501894{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-7f939532-7a64-4700-815d-65afeab75c63:NORMAL:10.216.2.22:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,917 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.22:50010 is added to blk_2880883147_1896501896{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-7f939532-7a64-4700-815d-65afeab75c63:NORMAL:10.216.2.22:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,919 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883151_1896501900{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|RBW]]} size 0
2019-03-12 16:20:43,919 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.3:50010 is added to blk_2880883152_1896501901{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-bffbb1b4-09d5-486c-8493-13953032e792:NORMAL:10.216.5.3:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,924 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.5:50010 is added to blk_2880883158_1896501907{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,936 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883160_1896501909{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,947 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.3:50010 is added to blk_2880883164_1896501913{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-bffbb1b4-09d5-486c-8493-13953032e792:NORMAL:10.216.5.3:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.20:50010 is added to blk_2880883165_1896501914{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-32001683-8bf1-4ae1-baf4-0efd7d117c94:NORMAL:10.216.2.20:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.21:50010 is added to blk_2880883165_1896501914{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-32001683-8bf1-4ae1-baf4-0efd7d117c94:NORMAL:10.216.2.20:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.48:50010 is added to blk_2880883166_1896501915{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-cc0c9e65-7223-4a96-9066-544d069b4f7c:NORMAL:10.216.4.48:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.23:50010 is added to blk_2880883167_1896501916{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68b7e245-3c92-4f81-95fe-dcb87e2453a8:NORMAL:10.216.2.23:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,960 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.38:50010 is added to blk_2880883169_1896501918{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9dec2025-66f8-4b4a-a197-e56751b284d5:NORMAL:10.216.4.38:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,960 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.39:50010 is added to blk_2880883171_1896501920{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-645de8ff-8245-4bc8-b21b-0a1f05190d90:NORMAL:10.216.2.39:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,960 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.9:50010 is added to blk_2880883179_1896501928{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9056dc2d-4567-4539-991b-f0c24bcf190d:NORMAL:10.216.2.9:50010|FINALIZED]]} size 0
也可以通过HDFS
的Web UI
的startup
界面看到NameNode
加载元数据的进度,当startup
页面显示100%
时,表明NameNode
加载元数据完成:
(2)启动Hive
服务:
#启动hiveserver2
nohup hive --service hiveserver2 &
#启动metastore
nohup hive --service metastore &
5、启动resourcemanager
6、去掉namenode01
里的exclude
配置
1、通知运维修改服务器主机名称,并更新host
:
namenode01.bi(old) -> datanode19.bi
namenode02.bi -> datanode20.bi
2、在/usr/local/hadoop-2.6.3/etc/hadoop/dfs.include
文件中添加新节点名称:
datanode19.bi
datanode20.bi
3、在hdfs-site.xml
文件中添加属性:
<property>
<name>dfs.hosts</name>
<value>/usr/local/hadoop-2.6.3/etc/hadoop/dfs.include</value>
</property>
4、在NN
上刷新节点,不需要重启NN
服务:
hdfs dfsadmin -refreshNodes
5、在slaves
文件中添加新节点主机名,并同步更新到所有集群服务器:
datanode19.bi
datanode20.bi
6、单独在新节点的机器上启动新节点上的DataNode
:
/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh start datanode