首页
学习
活动
专区
圈层
工具
发布
首页
学习
活动
专区
圈层
工具
MCP广场
社区首页 >问答首页 >java异常":org.apache.hadoop.ipc.RpcException: RPC响应超过最大数据长度“

java异常":org.apache.hadoop.ipc.RpcException: RPC响应超过最大数据长度“
EN

Stack Overflow用户
提问于 2018-03-02 07:14:10
回答 2查看 8.3K关注 0票数 2

我正在尝试从运行在我桌面的Eclipse中的Java程序连接到远程HDFS。我能够连接。但在尝试读取数据时会出现以下异常:

原因: org.apache.hadoop.ipc.RpcException: RPC响应超过最大数据

有人能帮个忙吗?

我有一个非常基本的代码,用于读取来自hdfs.open()的测试data.Error;

代码语言:javascript
运行
复制
FileSystem hdfs =null;
    String uriPath = "hdfs://" + Constants.HOST + ":" + Constants.PORT+ "/test/hello_world.txt";
    String hadoopBase ="hdfs://" + Constants.HOST + ":" + Constants.PORT;
    Configuration conf = new Configuration();
    conf.set("fs.default.name", hadoopBase);
    URI uri;
    InputStream inputStream = null;
    try {
        uri = new URI(uriPath);
        hdfs =  FileSystem.get(uri, conf);
        Path path = new Path(uri);
        inputStream = hdfs.open(path);
        IOUtils.copyBytes(inputStream, System.out, 4096, false);
    } catch (URISyntaxException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } finally {
        try {
            hdfs.close();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        IOUtils.closeStream(inputStream);
    }

下面是完整的例外:

代码语言:javascript
运行
复制
java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:785)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1485)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1337)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
at com.sun.proxy.$Proxy11.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:826)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:815)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:804)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:319)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:281)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1115)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:321)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:333)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:786)
at DataUtil.readData(DataUtil.java:29)
at main(Main.java:24)
Caused by: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length
at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1800)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1155)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1052)
EN

回答 2

Stack Overflow用户

发布于 2020-03-16 15:36:53

检查您的core-site.xml

代码语言:javascript
运行
复制
<property>
    <name>fs.default.name</name>
    <value>hdfs://host:port</value>
</property>

此端口可以是90008020。确保在代码或命令中使用相同的端口

票数 6
EN

Stack Overflow用户

发布于 2018-05-23 17:14:32

尝试此解决方案:将此配置添加到hdfs-site.xml

代码语言:javascript
运行
复制
<property>
     <name>ipc.maximum.data.length</name>
     <value>134217728</value>
</property>
票数 1
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/49060244

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档