专栏首页牛肉圆粉不加葱org.apache.spark.streaming.dstream.MappedDStream@5a69b104 has not been initialized现象原因及方案

org.apache.spark.streaming.dstream.MappedDStream@5a69b104 has not been initialized现象原因及方案

现象

执行Spark Streaming Application时报错

15/07/09 11:26:55 INFO scheduler.JobGenerator: Stopping JobGenerator immediately
15/07/09 11:26:55 INFO util.RecurringTimer: Stopped timer for JobGenerator after time -1
15/07/09 11:26:55 INFO streaming.CheckpointWriter: CheckpointWriter executor terminated ? true, waited for 0 ms.
15/07/09 11:26:55 INFO scheduler.JobGenerator: Stopped JobGenerator
15/07/09 11:26:55 INFO scheduler.JobScheduler: Stopped JobScheduler
Exception in thread "main" org.apache.spark.SparkException: org.apache.spark.streaming.dstream.MappedDStream@5a69b104 has not been initialized
    at org.apache.spark.streaming.dstream.DStream.isTimeValid(DStream.scala:321)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:342)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:342)
    at scala.Option.orElse(Option.scala:257)
    at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:339)
    at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:38)
    at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:120)
    at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:120)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
    at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:120)
    at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$restart$4.apply(JobGenerator.scala:227)
    at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$restart$4.apply(JobGenerator.scala:222)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at org.apache.spark.streaming.scheduler.JobGenerator.restart(JobGenerator.scala:222)
    at org.apache.spark.streaming.scheduler.JobGenerator.start(JobGenerator.scala:92)
    at org.apache.spark.streaming.scheduler.JobScheduler.start(JobScheduler.scala:73)
    at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:588)
    at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:586)
    at DubaNewsMinisiteClick$.main(DubaNewsMinisiteClick.scala:171)
    at DubaNewsMinisiteClick.main(DubaNewsMinisiteClick.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

原因及方案

  • 原因: StreamingContext.checkpoint(...)指定的checkpoint目录下有另一个application生成的checkpoint文件, 导致StreamContext.getOrCreate时加载checkpoint失败, StreamingContext初始化失败
  • 解决方案: 删除该checkpoint目录或换一个目录进行checkpoint。 对于重新编译的Application, checkpoint目录下必须没有其他application生成的checkpoint文件

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

    在spark-shell中执行streaming application时,频繁出现以下错误。但是相同的代码在之前执行成功并无任务错误,集群以及spark的配置...

    codingforfun
  • [源码剖析]Spark读取配置Spark读取配置

    我们知道,有一些配置可以在多个地方配置。以配置executor的memory为例,有以下三种方式:

    codingforfun
  • (3) - Scala case class那些你不知道的知识

    除了在模式匹配中使用之外,unapply 方法可以让你结构 case class 来提取它的字段,如:

    codingforfun
  • spark on yarn 配置及异常解决

    转载请务必注明原创地址为:http://dongkelun.com/2018/04/16/sparkOnYarnConf/

    董可伦
  • spark on yarn 配置及异常解决

    转载请务必注明原创地址为:http://dongkelun.com/2018/04/16/sparkOnYarnConf/

    董可伦
  • Spark读取CSV异常 java.lang.ArrayIndexOutOfBoundsException:62

    情况1: 将GBK编码的文件转文UTF-8(我碰见的),当然这种情况也可以用情况2中的解决办法解决~

    董可伦
  • Spark No FileSystem for scheme file 解决方法

    这里的 Local repository 就是项目保存库的位置。在这里面依次打开文件位置:

    机器学习和大数据挖掘
  • Spark2.3.1+Kafka0.9使用Direct模式消费信息异常

    在验证kafka属性时不能使用scala默认的类,需要指定kafka带的类 createDirectStream[String, String, StringD...

    笨兔儿
  • spark1.4加载mysql数据 创建Dataframe及join操作连接方法问题

    最后无奈。。就用原来的方法 创建软连接,加载数据,发现可以。。这我就不明白了。。。

    用户3003813
  • org.apache.spark.sql.AnalysisException: Table or view not found: `traintext`.`train`; line 1 pos 14;

    恭喜老铁,跟我遇到了一样的问题,接下来是解决方法: 遇到的问题: org.apache.spark.sql.AnalysisException: Table o...

    用户1171305

扫码关注云+社区

领取腾讯云代金券