前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >Hadoop NameNode、DataNode热迁移方案

Hadoop NameNode、DataNode热迁移方案

作者头像
九州暮云
发布2019-08-21 10:38:55
1.9K0
发布2019-08-21 10:38:55
举报
文章被收录于专栏:九州牧云九州牧云九州牧云

最近我们生产环境的Hadoop集群需要调整几台服务器,具体转换关系如下:

datanode92.bi -> namenode02.bi
namenode01.bi(old) -> datanode19.bi
namenode02.bi -> datanode20.bi

最终目标为:

  • datanode92.binamenode01.bi服务器上的DataNode服务下线
  • 由于namenode02.bi服务器配置较低,因此将namenode02.bi服务器上的NameNode服务下线,将其迁移到配置更高的datanode92.bi服务器上
  • 利用闲置资源,将老的空闲的namenode01.bi(old)服务器调整为DataNode服务器
  • 调整完后,增加datanode19.bidatanode20.bi两个服务器名称

一、DataNode下线

1、在namenode01上,添加退役节点的IP到黑名单,/usr/local/hadoop-2.6.3/etc/hadoop/dfs.exclude文件添加如下需要下线的服务器主机名称:

datanode92.bi
namenode01.bi

2、配置hdfs-site.xml

<property>
    <name>dfs.hosts.exclude</name>
    <value>/usr/local/hadoop-2.6.3/etc/hadoop/dfs.exclude</value>
</property>

3、切换到/usr/local/hadoop-2.6.3/bin目录下,执行如下命令,刷新NN节点,不需要重启NN服务:

./hdfs dfsadmin -refreshNodes

执行命令后,需要及时查看NameNode日志验证命令是否执行成功,hadoop-hadoop-namenode-namenode01.bi.log输出日志如下:

Refresh nodes successful for namenode01.bi.10101111.com/10.216.2.25:8020
Refresh nodes successful for namenode02.bi.10101111.com/10.216.5.2:8020

4、查看spaceX,观察状态在decommisstion in progress的节点:

同时,还要查看需要进行拷贝的block副本数,也就是Number of Under-Replicated Blocks这个指标的值:

5、当所有要退役的节点状态都为Decommissioned,且Number of Under-Replicated Blocks数值为0,表明数据迁移工作已经完成

6、从slaves文件中清空退役节点

7、DN下线完成后,spaceX报磁盘满的警告,经过查看整个集群空间使用量及剩余量信息,证实可以忽略此警告:

(1)警告信息:

(2)磁盘剩余量17.46%

二、NameNode迁移

1、停止服务:

  • 停止namenode02.bi上的 namenodezkfcjournalnoderesourcemanager 服务,使namenode02.bi上的修改记录数据和元数据不再更新:
/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh stop namenode
/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh stop zkfc
/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh stop journalnode
  • 停止Hive服务:查出hiveserver2metastore的进程号,直接kill

2、通知运维更新网络信息:

  • 更新DNS信息
  • 修改主机名称,datanode92.bi的主机名称修改为namenode02.bi的主机名称,并同步更新Hadoop集群服务器的host文件、slaves文件

3、拷贝数据:

  • NameNode的整个Hadoop目录拷贝到目标机器的/usr/local路径下
  • /data/dfs的元数据拷贝到目标机器的/data/dfs路径下
scp -r dfs hadoop@datanode92.bi:/data/dfs/
  • /usr/local/apache-hive-1.2.1-bin拷贝到目标机器的/usr/local路径下

4、启动服务:

(1)检查配置文件,在目标机器上启动namenodezkfcjournalnode服务。以下是NameNode服务启动后的块汇报日志,可以看到大量带有BlockStateChange关键字的日志,这样的日志打印没完成,就证明NameNode没有完全正常,不能对外提供服务,日志如下:

2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.26:50010 is added to blk_2880883080_1896501829{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2297011b-a18a-449b-b795-248b44d082eb:NORMAL:10.216.5.26:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.21:50010 is added to blk_2880883080_1896501829{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2297011b-a18a-449b-b795-248b44d082eb:NORMAL:10.216.5.26:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.40:50010 is added to blk_2880883080_1896501829{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2297011b-a18a-449b-b795-248b44d082eb:NORMAL:10.216.5.26:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-0e5e6e27-8744-4e6a-8006-690f6e099676:NORMAL:10.216.5.40:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.21:50010 is added to blk_2880883081_1896501830{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.4:50010 is added to blk_2880883081_1896501830{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-fa7fa32c-91bb-47a3-a559-2fc31f74cdec:NORMAL:10.216.4.4:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.48:50010 is added to blk_2880883081_1896501830{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-fa7fa32c-91bb-47a3-a559-2fc31f74cdec:NORMAL:10.216.4.4:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-df7f2d4a-0b76-46ee-9e30-d42921ba825a:NORMAL:10.216.5.48:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.21:50010 is added to blk_2880883082_1896501831{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.19:50010 is added to blk_2880883082_1896501831{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-2c37fdd9-9d89-493c-bb0d-b915b9fd0476:NORMAL:10.216.4.19:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,795 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.41:50010 is added to blk_2880883082_1896501831{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e5c6c2d5-81ae-45ee-a204-f3e3313756fe:NORMAL:10.216.4.21:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-2c37fdd9-9d89-493c-bb0d-b915b9fd0476:NORMAL:10.216.4.19:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-91a83f26-10a5-4456-94ec-460d5a428990:NORMAL:10.216.4.41:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,796 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.34:50010 is added to blk_2880883083_1896501832{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-5540b823-7606-432a-98c9-a8d84c2c3e72:NORMAL:10.216.2.34:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,796 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.8:50010 is added to blk_2880883083_1896501832{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-5540b823-7606-432a-98c9-a8d84c2c3e72:NORMAL:10.216.2.34:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-110ef02a-b044-41bf-9d65-f5fdaa76b959:NORMAL:10.216.2.8:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,796 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.8:50010 is added to blk_2880883083_1896501832{blockUCState=UNDER_CONSTRUCTION, prima849 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883130_1896501879{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,863 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log: 132651/150565 transactions completed. (88%)
2019-03-12 16:20:43,879 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.38:50010 is added to blk_2880883136_1896501885{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9dec2025-66f8-4b4a-a197-e56751b284d5:NORMAL:10.216.4.38:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,893 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.5:50010 is added to blk_2880883140_1896501889{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,893 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.16:50010 is added to blk_2880883140_1896501889{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-32c95765-9f02-4863-8ba2-f1fecf2e0872:NORMAL:10.216.5.16:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,893 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.20:50010 is added to blk_2880883140_1896501889{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-32c95765-9f02-4863-8ba2-f1fecf2e0872:NORMAL:10.216.5.16:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-32001683-8bf1-4ae1-baf4-0efd7d117c94:NORMAL:10.216.2.20:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,904 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.38:50010 is added to blk_2880883143_1896501892{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9dec2025-66f8-4b4a-a197-e56751b284d5:NORMAL:10.216.4.38:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,905 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.23:50010 is added to blk_2880883144_1896501893{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68b7e245-3c92-4f81-95fe-dcb87e2453a8:NORMAL:10.216.2.23:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,905 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.22:50010 is added to blk_2880883144_1896501893{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68b7e245-3c92-4f81-95fe-dcb87e2453a8:NORMAL:10.216.2.23:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-6b7b44fb-539f-4a1c-a867-1a0d7fe77972:NORMAL:10.216.5.22:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,912 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883145_1896501894{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,912 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.22:50010 is added to blk_2880883145_1896501894{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-7f939532-7a64-4700-815d-65afeab75c63:NORMAL:10.216.2.22:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,917 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.22:50010 is added to blk_2880883147_1896501896{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-7f939532-7a64-4700-815d-65afeab75c63:NORMAL:10.216.2.22:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,919 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883151_1896501900{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|RBW]]} size 0
2019-03-12 16:20:43,919 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.3:50010 is added to blk_2880883152_1896501901{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-bffbb1b4-09d5-486c-8493-13953032e792:NORMAL:10.216.5.3:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,924 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.5:50010 is added to blk_2880883158_1896501907{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-45cbbbda-e17f-4835-bf47-7487c3b3a456:NORMAL:10.216.5.5:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,936 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.14:50010 is added to blk_2880883160_1896501909{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e80ba307-0231-4811-92af-33fac386ea09:NORMAL:10.216.5.14:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,947 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.5.3:50010 is added to blk_2880883164_1896501913{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-bffbb1b4-09d5-486c-8493-13953032e792:NORMAL:10.216.5.3:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.20:50010 is added to blk_2880883165_1896501914{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-32001683-8bf1-4ae1-baf4-0efd7d117c94:NORMAL:10.216.2.20:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.21:50010 is added to blk_2880883165_1896501914{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-32001683-8bf1-4ae1-baf4-0efd7d117c94:NORMAL:10.216.2.20:50010|FINALIZED], ReplicaUnderConstruction[[DISK]DS-b4850b87-338c-45cc-b917-b6250aa70370:NORMAL:10.216.2.21:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.48:50010 is added to blk_2880883166_1896501915{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-cc0c9e65-7223-4a96-9066-544d069b4f7c:NORMAL:10.216.4.48:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,957 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.23:50010 is added to blk_2880883167_1896501916{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68b7e245-3c92-4f81-95fe-dcb87e2453a8:NORMAL:10.216.2.23:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,960 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.4.38:50010 is added to blk_2880883169_1896501918{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9dec2025-66f8-4b4a-a197-e56751b284d5:NORMAL:10.216.4.38:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,960 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.39:50010 is added to blk_2880883171_1896501920{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-645de8ff-8245-4bc8-b21b-0a1f05190d90:NORMAL:10.216.2.39:50010|FINALIZED]]} size 0
2019-03-12 16:20:43,960 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 10.216.2.9:50010 is added to blk_2880883179_1896501928{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9056dc2d-4567-4539-991b-f0c24bcf190d:NORMAL:10.216.2.9:50010|FINALIZED]]} size 0

也可以通过HDFSWeb UIstartup界面看到NameNode加载元数据的进度,当startup页面显示100%时,表明NameNode加载元数据完成:

(2)启动Hive服务:

#启动hiveserver2
nohup hive --service hiveserver2 &
#启动metastore
nohup hive --service metastore &

5、启动resourcemanager

6、去掉namenode01里的exclude配置

三、DataNode上线

1、通知运维修改服务器主机名称,并更新host

namenode01.bi(old) -> datanode19.bi
namenode02.bi -> datanode20.bi

2、在/usr/local/hadoop-2.6.3/etc/hadoop/dfs.include文件中添加新节点名称:

datanode19.bi
datanode20.bi

3、在hdfs-site.xml文件中添加属性:

<property>
    <name>dfs.hosts</name>
    <value>/usr/local/hadoop-2.6.3/etc/hadoop/dfs.include</value>
</property>

4、在NN上刷新节点,不需要重启NN服务:

hdfs dfsadmin -refreshNodes

5、在slaves文件中添加新节点主机名,并同步更新到所有集群服务器:

datanode19.bi
datanode20.bi

6、单独在新节点的机器上启动新节点上的DataNode

/usr/local/hadoop-2.6.3/sbin/hadoop-daemon.sh start datanode

四、问题记录及参考资料

NameNode热迁移方案

如何在不影响hadoop集群正常运行的情况下迁移主控节点[namenode]

Jps介绍以及解决jps无法查看某个已经启动的java进程问题

本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • 一、DataNode下线
  • 二、NameNode迁移
  • 三、DataNode上线
  • 四、问题记录及参考资料
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档