java.io.IOException:帧大小[...]大于最大长度[...]！
java.io.IOException:帧大小[...]大于最大长度[...]！
提问于 2017-03-23 21:43:31
我在独立模式下运行Spark + Alluxio进行数据访问。更具体地说，我有一个火花大师和一个火花工作者。
当运行我的作业时，我得到以下错误：
17/03/22 14:35:43 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.254.22.6): java.io.IOException: Frame size (67108864) larger than max length (16777216)!
at alluxio.AbstractClient.checkVersion(AbstractClient.java:112)
at alluxio.AbstractClient.connect(AbstractClient.java:175)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
at alluxio.client.file.FileSystemMasterClient.getStatus(FileSystemMasterClient.java:183)
at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:175)
at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:167)
at alluxio.hadoop.HdfsFileInputStream.<init>(HdfsFileInputStream.java:86)
at alluxio.hadoop.AbstractFileSystem.open(AbstractFileSystem.java:514)
at alluxio.hadoop.FileSystem.open(FileSystem.java:25)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at net.atos.hadoop.ImageRecordReader.initialize(ImageRecordReader.java:47)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:153)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:124)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:65)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:262)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)环境：
- Spark 1.5.2
- Alluxio 1.3.0
回答 1
Stack Overflow用户
发布于 2017-03-23 21:44:12
我在master和worker上都将SPARK_WORKER_MEMORY设置为2G (将其作为环境变量传递，请参阅this section of the documentation)。我试着把它增加到4G。但是，我只更改了worker的参数。我猜这导致了master和worker之间的不匹配。
在两个节点上将其设置为相同的值(4G)解决了问题。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/42977830
复制相关文章
相似问题