创建SparkContext时,Pyspark会抛出JNI错误

内容来源于 Stack Overflow,并遵循CC BY-SA 3.0许可协议进行翻译与使用

  • 回答 (1)
  • 关注 (0)
  • 查看 (430)

我(尝试)使用Python2在Manjaro Linux上运行pyspark。我创建了一个测试脚本来创建一个SparkContext实例并再次停止它:

import findspark
findspark.init()
from pyspark import SparkContext

if __name__ == '__main__':
    sc = SparkContext()
    sc.stop()

我从一个终端启动它python2 filename.py。这在以前有效,但由于我不明白的原因,现在提出以下内容:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more
Traceback (most recent call last):
  File "mwe.py", line 22, in <module>
    sc = SparkContext()
  File "/opt/apache-spark/python/pyspark/context.py", line 115, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/apache-spark/python/pyspark/context.py", line 292, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/apache-spark/python/pyspark/java_gateway.py", line 93, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

我已经读过关于其他人使用Java版本8而不是9或10来解决这个问题。但是,我似乎已经在运行版本8,作为archlinux-java status输出:

Available Java environments:
  java-10-openjdk
  java-8-jdk
  java-8-jre/jre
  java-8-openjdk/jre (default)

我不知道如何从这里开始,所以任何帮助将不胜感激。

提问于
用户回答回答于

我有同样的问题,我通过卸载修复 java-10-openjdk

扫码关注云+社区

领取腾讯云代金券