我很难让PySpark在我的电脑上工作。我使用的是Python 3.7和Spark 2.4。我的代码如下:
import pyspark
from pyspark import SparkContext
sc = SparkContext(appName="Pyspark mllib Example")
'cmd' is not recognized as an internal or external command,
operable program or batch file.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\context.py", line 133, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\context.py", line 316, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\java_gateway.py", line 46, in launch_gateway
return _launch_gateway(conf)
File "C:\Users\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\java_gateway.py", line 108, in _launch_gateway
raise Exception("Java gateway process exited before sending its port number")
**Exception: Java gateway process exited before sending its port number**我收到一个错误:
Exception: Java gateway process exited before sending its port number发布于 2021-11-16 15:02:03
我也有同样的问题。事实证明,这是关于Java的路径是不正确的。
我做了以下工作:
nano ~/.zshrc将关于JAVA_HOME的行更改为:
export JAVA_HOME="/opt/homebrew/Cellar/openjdk@11/11.0.12"我还使用了一个虚拟环境来使用pyspark、spark和tutti quanti的正确版本。
https://stackoverflow.com/questions/60146948
复制相似问题