Pyspark在创建SparkContext时抛出JNI错误
Pyspark在创建SparkContext时抛出JNI错误
提问于 2019-01-06 00:29:58
我(尝试)在带Python2的Manjaro Linux上运行pyspark。我已经创建了一个测试脚本来创建一个SparkContext实例并再次停止它：
import findspark
findspark.init()
from pyspark import SparkContext
if __name__ == '__main__':
sc = SparkContext()
sc.stop()我使用python2 filename.py从终端启动此程序。这在以前是有效的，但由于我不理解的原因，现在引发了以下问题：
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
Traceback (most recent call last):
File "mwe.py", line 22, in <module>
sc = SparkContext()
File "/opt/apache-spark/python/pyspark/context.py", line 115, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/opt/apache-spark/python/pyspark/context.py", line 292, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/opt/apache-spark/python/pyspark/java_gateway.py", line 93, in launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number我已经读到了关于其他人使用Java8版本而不是9或10来解决这个问题的文章。然而，我似乎已经运行了版本8，作为archlinux-java status输出：
Available Java environments:
java-10-openjdk
java-8-jdk
java-8-jre/jre
java-8-openjdk/jre (default)我不知道如何从这里开始，所以任何帮助都会非常感谢。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/54053925
复制相关文章
相似问题