文档建议反馈控制台
首页
学习
活动
专区
圈层
工具
MCP广场
文章/答案/技术大牛
发布
首页
学习
活动
专区
圈层
工具
MCP广场
返回腾讯云官网
社区首页 >问答首页 >未能将数据同步到hadoop集群(未能放置足够多的副本)

问未能将数据同步到hadoop集群(未能放置足够多的副本)
EN

Stack Overflow用户
提问于 2022-06-09 04:29:41
回答 1查看 411关注 0票数 0

我在ubuntu20.04中安装了Hadoop集群。在安装Hadoop集群之后，我遇到了一个大问题。从客户端到Hadoop集群的数据同步管道已经中断。我打算传输到Hadoop集群的数据大约是18 to。虽然我剩下的DFS是65%，但是从客户端到Hadoop集群的管道传输文件失败了。我试图格式化nameNode和dataNode，但结果仍然失败。有人能帮我解决吗？以下是给我们的一些信息：

日志失败：

代码语言：javascript
运行
复制
2022-06-09 10:46:08,507 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not enough replicas was chosen. Reason: {NOT_ENOUGH_STORAGE_SPACE=3, NO_REQUIRED_STORAGE_TYPE=1}
2022-06-09 10:46:08,507 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not enough replicas was chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1}
2022-06-09 10:46:08,508 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 3 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology
2022-06-09 10:46:08,508 WARN org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough replicas: expected size is 3 but only 0 storage types can be selected (replication=3, selected=[], unavailable=[DISK], removed=[DISK, DISK, DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
2022-06-09 10:46:08,508 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 3 to reach 3 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable:  unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
2022-06-09 10:46:08,511 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on default port 9000, call Call#1744126 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.24.217.203:38560
java.io.IOException: File /.reserved/.inodes/30284/.9DFE31A1-742F11EC-981DADC9-CEAF568F@172.24.219.148.wav.C5j1SD could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2315)
    at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2960)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:904)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:593)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)
    at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)
    at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)

我的DFS管理报告：

代码语言：javascript
运行
复制
Name: 172.24.217.204:9866 (hadoop-node1)
Hostname: hadoop-node1
Decommission Status : Normal
Configured Capacity: 104091082752 (96.94 GB)
DFS Used: 19833880576 (18.47 GB)
Non DFS Used: 11562889216 (10.77 GB)
DFS Remaining: 67362725888 (62.74 GB)
DFS Used%: 19.05%
DFS Remaining%: 64.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Thu Jun 09 11:26:29 ICT 2022
Last Block Report: Thu Jun 09 09:54:17 ICT 2022
Num of Blocks: 9033


Name: 172.24.217.205:9866 (hadoop-node2)
Hostname: hadoop-node2
Decommission Status : Normal
Configured Capacity: 104091082752 (96.94 GB)
DFS Used: 19833790464 (18.47 GB)
Non DFS Used: 11416346624 (10.63 GB)
DFS Remaining: 67509358592 (62.87 GB)
DFS Used%: 19.05%
DFS Remaining%: 64.86%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Thu Jun 09 11:26:29 ICT 2022
Last Block Report: Thu Jun 09 10:02:23 ICT 2022
Num of Blocks: 9033


Name: 172.24.217.206:9866 (hadoop-node3)
Hostname: hadoop-node3
Decommission Status : Normal
Configured Capacity: 104091082752 (96.94 GB)
DFS Used: 19833802752 (18.47 GB)
Non DFS Used: 10835709952 (10.09 GB)
DFS Remaining: 68089982976 (63.41 GB)
DFS Used%: 19.05%
DFS Remaining%: 65.41%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Thu Jun 09 11:26:29 ICT 2022
Last Block Report: Thu Jun 09 09:54:17 ICT 2022
Num of Blocks: 9033

**我的核心-site.xml配置：

代码语言：javascript
运行
复制
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://hadoop-master:9000</value>
    </property>

<!-->HTTP/dfs proxy-->  
   <property>
      <name>hadoop.proxyuser.root.groups</name>
      <value>*</value>
   </property>
   <property>
      <name>hadoop.proxyuser.root.hosts</name>
      <value>*</value>
   </property>
</configuration>

我的hdfs-site.xml配置：

代码语言：javascript
运行
复制
 *<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/hadoop/dfsdata/nameNode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>[DISK]file:///opt/hadoop/dfsdata/dataNode</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.namenode.redundancy.considerLoad</name>
        <value>false</value>
    </property>
    
    <!-- HDFS NFS gateway -->
    <property>
        <name>dfs.namenode.accesstime.precision</name>
        <value>3600000</value>
        <description>
    The access time for HDFS file is precise up to this value. 
    The default value is 1 hour. Setting a value of 0 disables
    access times for HDFS.
  </description>
    </property>
    <property>
        <name>dfs.storage.policy.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.nfs3.dump.dir</name>
        <value>/tmp/.hdfs-nfs</value>
    </property>
    <property>
        <name>dfs.nfs.exports.allowed.hosts</name>
        <value>172.24.217.0/24 rw ; 172.30.12.0/24 rw ; 172.24.216.0/24 rw</value>
    </property>
    <property>
        <name>nfs.export.point</name>
        <value>/</value>
    </property>
    <property>
        <name>dfs.namenode.redundancy.considerLoad.factor</name>
        <value>3</value>
    </property>
</configuration>*
hadoop
EN

回答 1

Stack Overflow用户

发布于 2022-08-31 05:55:48

我使用了三台dev虚拟机来部署hadoop集群，并遇到了与您类似的问题。

我打开日志并根据提示设置调试级别org.apache.hadoop.net.NetworkTopology。

代码语言：javascript
运行
复制
2022-08-31 10:43:32,576 DEBUG org.apache.hadoop.net.NetworkTopology: Choosing random from 2 available nodes on node /default-rack, scope=/default-rack, excludedScope=null, excludeNodes=[]. numOfDatanodes=2.
2022-08-31 10:43:32,576 DEBUG org.apache.hadoop.net.NetworkTopology: chooseRandom returning x.x.x.x:9866
2022-08-31 10:43:32,577 DEBUG org.apache.hadoop.net.NetworkTopology: Failed to find datanode (scope="" excludedScope="/default-rack"). numOfDatanodes=0
2022-08-31 10:43:32,577 DEBUG org.apache.hadoop.net.NetworkTopology: No node to choose.

我怀疑这是网络问题。我去那台机器打电话给9866端口，结果它真的不起作用了。然后我打开了这个港口。打开这个端口后，我的hadoop问题就解决了。

我希望这对你有帮助。

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：

https://stackoverflow.com/questions/72554747

复制
相关文章

相似问题

领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2025 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 深公网安备号 44030502008569

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号 | 京公网安备号11010802020287

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档