python下的pyspark报错集锦
python下的pyspark报错集锦
1.WARN Utils: Service ‘SparkUI’ could not bind on port 4040. Attempting port 4041.
出现这种错误是是在spark启动从节点时出现的。 解决的方法是，在spark-env.sh中加入一条 SPARK_LOCAL_IP=127.0.0.1 然后就完美解决报错了！ D:\spark\spark-2.2.0-bin-hadoop2.7\bin找到load-spark-env.sh,之后notepad打开，增加如下，完成
2.WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/D:/spark/spark-2.2.0-bin-hadoop2.7/jars/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use –illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 18/05/11 17:07:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
出现以上错误，由于JDK版本为9，太高了，但是不影响运行。可以无事
3.ython in worker has different version 3.6 than that in driver 3.5, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
问题解决：
import os
os.environ["PYSPARK_PYTHON"]="D:\office3\python\\anaconda3.5\\3.5\envs\python35\\python"指定运行的python环境位置。