dataframe内连接中出现Pyspark awaitResult错误
dataframe内连接中出现Pyspark awaitResult错误
提问于 2018-05-22 02:43:05
在docker容器中运行独立的spark-2.3.0-bin-hadoop2.7
- df1 =5行
- df2= 10行
- 数据集非常小。
df1 schema: Dataframe[id:bigint, name:string] df2 schema: Dataframe[id:decimal(12,0), age: int]
内连接
df3 = df1.join(df2, df1.id == df2.id, 'inner')
df3 schema: Dataframe[id:bigint, name:string, age: int]执行df3.show(5)时，出现以下错误
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/apache/spark-2.3.0-bin-hadoop2.7/python/pyspark/sql/dataframe.py", line 466, in collect
port = self._jdf.collectToPython() File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name) File "/usr/apache/spark-2.3.0-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw) File "/usr/local/lib/python3.6/dist-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value) py4j.protocol.Py4JJavaError: An error occurred while calling o43.collectToPython. : org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:136)已尝试按照this suggestion将广播超时设置为-1，但得到相同的错误
conf = SparkConf().set("spark.sql.broadcastTimeout","-1")回答 1
Stack Overflow用户
回答已采纳
发布于 2018-06-07 04:39:55
我使用的是与Spark 2.3不兼容的JRE版本。
在Docker Image中使用openjdk-8-jre更新JRE后解决了错误
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/50454850
复制相关文章
相似问题