我(尝试)在带Python2的Manjaro Linux上运行pyspark。我已经创建了一个测试脚本来创建一个SparkContext实例并再次停止它:
import findspark
findspark.init()
from pyspark import SparkContext
if __name__ == '__main__':
sc = SparkContext()
sc.stop()
我使用python2 filename.py
从终端启动此程序。这在以前是有效的,但由于我不理解的原因,现在引发了以下问题:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
Traceback (most recent call last):
File "mwe.py", line 22, in <module>
sc = SparkContext()
File "/opt/apache-spark/python/pyspark/context.py", line 115, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/opt/apache-spark/python/pyspark/context.py", line 292, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/opt/apache-spark/python/pyspark/java_gateway.py", line 93, in launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
我已经读到了关于其他人使用Java8版本而不是9或10来解决这个问题的文章。然而,我似乎已经运行了版本8,作为archlinux-java status
输出:
Available Java environments:
java-10-openjdk
java-8-jdk
java-8-jre/jre
java-8-openjdk/jre (default)
我不知道如何从这里开始,所以任何帮助都会非常感谢。
发布于 2019-03-12 04:38:54
我也遇到了同样的问题,我通过卸载java-10-openjdk
修复了这个问题
https://stackoverflow.com/questions/54053925
复制相似问题