“已建立的连接已被主机中的软件中止”,同时从Java代码将文件复制到Hadoop HDFS

内容来源于 Stack Overflow,并遵循CC BY-SA 3.0许可协议进行翻译与使用

  • 回答 (2)
  • 关注 (0)
  • 查看 (99)

我正在尝试一些简单的Java代码将文件从本地系统复制到HDFS。这是我的简单方法的样子:

private static void copyFileToHDFS() throws IOException
{
    config.set("fs.defaultFS","hdfs://127.0.0.1:9000");
    FileSystem hdfs = FileSystem.get(config);
    Path localfsSourceDir = new Path("D:\\file1");
    Path hdfsTargetDir = new Path ("hdfs://127.0.0.1:9000/dir/");
    hdfs.copyFromLocalFile(localfsSourceDir, hdfsTargetDir);      //throws Exception
}

这给了我最后一行的Exception:

Exception in thread "main" java.io.IOException: Failed on local exception: java.io.IOException: An established connection was aborted by the software in your host machine; Host Details : local host is: "01hw713648/10.163.5.139"; destination host is: "127.0.0.1":9000; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
    at org.apache.hadoop.ipc.Client.call(Client.java:1479)
    at org.apache.hadoop.ipc.Client.call(Client.java:1412)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at $Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at $Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
    at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:496)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:348)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
    at HBaseImportTsvBulkLoader.copyFileToHDFS(HBaseImportTsvBulkLoader.java:64)
    at HBaseImportTsvBulkLoader.main(HBaseImportTsvBulkLoader.java:37)
Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(Unknown Source)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
    at sun.nio.ch.IOUtil.read(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
    at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at java.io.FilterInputStream.read(Unknown Source)
    at java.io.FilterInputStream.read(Unknown Source)
    at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:520)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at java.io.DataInputStream.readInt(Unknown Source)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1084)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:979)

我的设置

我在运行在VirtualBox上的Ubuntu VM中运行Hadoop集群(后者又在Windows上运行)。我的集群运行完美。我在Windows上运行这个java代码。我在Virtual Box上设置了端口转发规则,如下所示:

   Name                    Protocol      Host-ip          Host-port       Guest-Ip       Guest-port
   datanode                tcp           <left empty>     50075           <guest-ip>     50075 
   dfs web ui              tcp           <left empty>     50070           <guest-ip>     50070
   mapred apps             tcp           <left empty>     8088            <guest-ip>     8088
   hbase web ui            tcp           <left empty>     16010           <guest-ip>     16010
   hdfs                    tcp           <left empty>     9000            <guest-ip>     9000 
   regionserver web ui     tcp           <left empty>     16301           <guest-ip>     16301
   ssh                     tcp           <left empty>     22              <guest-ip>     22

这允许我连接到VM上的各种服务:

  • 我可以使用putty连接到VM
  • 也可以在Windows中的浏览器中打开各种hadoop web uis:Namenode web ui,HMaster webui,Regionserver webui

更新

一些类似错误但在Hadoop上下文中没有的在线解决方案建议重启eclipse。所以我试过了。现在得到略有不同的错误:

Exception in thread "main" java.io.IOException: Failed on local exception: java.io.IOException: An existing connection was forcibly closed by the remote host; Host Details : local host is: "01hw713648/10.163.5.139"; destination host is: "127.0.0.1":9000;

具有完全相同的堆栈跟踪。

提问于
用户回答回答于

当指定的端口错误时,我最近得到了这个错误。我指定了50070而不是9000。

我使用了仅限主机的适配器,因此不需要端口转发。我从Windows主机连接到Linux VM中的192.168.xx。我还确保名称节点正在核心site.xml中的0.0.0.0:9000上进行侦听。VM中禁用了防火墙和SE Linux。我现在正在设置工作。

用户回答回答于

得到了解决。我localhost在所有hadoop xyz-site.xml文件中指定。我把它们全都改成了<guest-vm-ip>

扫码关注云+社区

领取腾讯云代金券