blocks|key|1677495|text|原来pyspark+bin正在加载python并自动加载正确的库路径。查看$SPARK_HOME/bin/pyspark：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1677496|#+Add+the+PySpark+classes+to+the+Python+path:
export+PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH|code-block|syntax|javascript|1677497|我将这一行添加到我的.bashrc文件中，现在可以正确找到模块了！|1677498|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

Turns out that the pyspark bin is LOADING python and automatically loading the correct library paths. Check out $SPARK_HOME/bin/pyspark :

<pre><code># Add the PySpark classes to the Python path:
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
</code></pre>

I added this line to my .bashrc file and the modules are now correctly found!

blocks|key|1891422|text|不要将py文件作为：python+filename.py运行，而应使用：spark-submit+filename.py|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1891423|entityMap^0|A|I|10|O|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|M|8|@]|D|@]|E|$]]]|G|$]]

dont run your py file as: <code>python filename.py</code>
instead use: <code>spark-submit filename.py</code>

blocks|key|1677621|text|通过导出SPARK路径和Py4j路径，它开始工作：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1677622|export+SPARK_HOME=/usr/local/Cellar/apache-spark/1.5.1
export+PYTHONPATH=$SPARK_HOME/libexec/python:$SPARK_HOME/libexec/python/build:$PYTHONPATH
PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH+
export+PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/build:$PYTHONPATH|code-block|syntax|javascript|1677623|因此，如果您不想在每次启动Python+shell时都输入这些命令，则可能需要将其添加到.bashrc文件中|offset|length|style|CODE|1677624|entityMap^0|0|0|18|7|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@$I|R|J|S|K|L]]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

By exporting the SPARK path and the Py4j path, it started to work:

<pre><code>export SPARK_HOME=/usr/local/Cellar/apache-spark/1.5.1
export PYTHONPATH=$SPARK_HOME/libexec/python:$SPARK_HOME/libexec/python/build:$PYTHONPATH
PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH 
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/build:$PYTHONPATH
</code></pre>

So, if you don't want to type these everytime you want to fire up the Python shell, you might want to add it to your <code>.bashrc</code> file

blocks|key|1677549|text|在Mac上，我使用Homebrew安装Spark+(公式为"apache-spark")。然后，我这样设置PYTHONPATH，这样Python导入就可以工作了：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1677550|export+SPARK_HOME=/usr/local/Cellar/apache-spark/1.2.0
export+PYTHONPATH=$SPARK_HOME/libexec/python:$SPARK_HOME/libexec/python/build:$PYTHONPATH|code-block|syntax|javascript|1677551|将mac上的"1.2.0“替换为实际的apache-spark版本。|1677552|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

On Mac, I use Homebrew to install Spark (formula "apache-spark"). Then, I set the PYTHONPATH this way so the Python import works:

<pre><code>export SPARK_HOME=/usr/local/Cellar/apache-spark/1.2.0
export PYTHONPATH=$SPARK_HOME/libexec/python:$SPARK_HOME/libexec/python/build:$PYTHONPATH
</code></pre>

Replace the "1.2.0" with the actual apache-spark version on your mac.

blocks|key|1891909|text|要去掉ImportError:+No+module+named+py4j.java_gateway，需要添加以下几行：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1891910|import+os
import+sys


os.environ['SPARK_HOME']+=+"D:\python\spark-1.4.1-bin-hadoop2.4"


sys.path.append("D:\python\spark-1.4.1-bin-hadoop2.4\python")
sys.path.append("D:\python\spark-1.4.1-bin-hadoop2.4\python\lib\py4j-0.8.2.1-src.zip")

try:
++++from+pyspark+import+SparkContext
++++from+pyspark+import+SparkConf

++++print+("success")

except+ImportError+as+e:
++++print+("error+importing+spark+modules",+e)
++++sys.exit(1)|code-block|syntax|javascript|1891911|entityMap^0|3|1A|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|P|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|D|@]|E|$]]]|L|$]]

To get rid of <code>ImportError: No module named py4j.java_gateway</code>, you need to add following lines:

<pre><code>import os
import sys


os.environ['SPARK_HOME'] = "D:\python\spark-1.4.1-bin-hadoop2.4"


sys.path.append("D:\python\spark-1.4.1-bin-hadoop2.4\python")
sys.path.append("D:\python\spark-1.4.1-bin-hadoop2.4\python\lib\py4j-0.8.2.1-src.zip")

try:
 from pyspark import SparkContext
 from pyspark import SparkConf

 print ("success")

except ImportError as e:
 print ("error importing spark modules", e)
 sys.exit(1)
</code></pre>

blocks|key|1678126|text|在Windows10上，以下方法对我有效。我使用设置>编辑环境变量为您的帐户添加了以下环境变量|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1678127|SPARK_HOME=C:\Programming\spark-2.0.1-bin-hadoop2.7
PYTHONPATH=%25SPARK_HOME%25\python;%25PYTHONPATH%25|code-block|syntax|javascript|1678128|(更改"C:\Programming...“到您安装spark的文件夹)|1678129|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

On Windows 10 the following worked for me. I added the following environment variables using Settings > Edit environment variables for your account:

<pre><code>SPARK_HOME=C:\Programming\spark-2.0.1-bin-hadoop2.7
PYTHONPATH=%SPARK_HOME%\python;%PYTHONPATH%
</code></pre>

(change "C:\Programming\..." to the folder in which you have installed spark)

blocks|key|1678159|text|我正在CentOS+VM上运行spark集群，它是从cloudera+yum包安装的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1678160|必须设置以下变量才能运行pyspark。|1678161|export+SPARK_HOME=/usr/lib/spark;
export+PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH|code-block|syntax|javascript|1678162|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

I am running a spark cluster, on CentOS VM, which is installed from cloudera yum packages.

Had to set the following variables to run pyspark.

<pre><code>export SPARK_HOME=/usr/lib/spark;
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH
</code></pre>

blocks|key|1892004|text|export+PYSPARK_PYTHON=/home/user/anaconda3/bin/python
export+PYSPARK_DRIVER_PYTHON=jupyter
export+PYSPARK_DRIVER_PYTHON_OPTS='notebook'|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1892005|这是我在Spark中使用Anaconda发行版时所做的事情。这是独立于Spark版本的。您可以将第一行更改为用户的python+bin。此外，从Spark+2.2.0开始，PySpark在PyPi上是一个独立的软件包，但我还没有对其进行测试。|unstyled|1892006|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|J|8|@]|9|@]|A|$]]|$1|G|3|-4|5|F|7|K|8|@]|9|@]|A|$]]]|H|$]]

<pre><code>export PYSPARK_PYTHON=/home/user/anaconda3/bin/python
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
</code></pre>

This is what I did for using my Anaconda distribution with Spark.
This is Spark version independent.
You can change the first line to your users' python bin. 
Also, as of Spark 2.2.0 PySpark is available as a Stand-alone package on PyPi
but I am yet to test it out.

blocks|key|1678348|text|您可以使用pip获取python中的pyspark+path+(如果您使用PIP安装了pyspark+)，如下所示|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1678349|pip+show+pyspark|code-block|syntax|javascript|1678350|entityMap^0|5|3|I|C|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]|$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|S|8|@]|D|@]|E|$]]]|L|$]]

You can get the <code>pyspark path</code> in python using <code>pip</code> (if you have installed pyspark using PIP) as below

<pre><code>pip show pyspark
</code></pre>

blocks|key|1892040|text|我也有同样的问题。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1892041|还要确保您使用的是正确的python版本，并且您正在使用正确的pip版本安装它。在我的例子中:我同时使用了python+2.7和3.x。我已经安装了pyspark和|1892042|pip2.7安装pyspark|1892043|而且它起作用了。|1892044|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|K|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|L|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|M|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|I|$]]

I had the same problem.

Also make sure you are using right python version and you are installing it with right pip version. in my case: I had both python 2.7 and 3.x. 
I have installed pyspark with 

pip2.7 install pyspark

and it worked.

blocks|key|1678300|text|在我的例子中，它是在不同的python+dist_package+(Python3.5)上安装的，而我使用的是Python3.6，所以下面的帮助：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1678301|python+-m+pip+install+pyspark|code-block|syntax|javascript|1678302|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

In my case it was getting install at a different python dist_package (python 3.5) whereas I was using python 3.6,
so the below helped:

<pre><code>python -m pip install pyspark
</code></pre>

blocks|key|1891765|text|我得到这个错误是因为我试图提交的python脚本名为pyspark.py+(facepalm)。修复方法是按照上面的建议设置我的PYTHONPATH，然后将脚本重命名为pyspark_test.py，并清除基于我的脚本原始名称创建的pyspark.pyc并清除此错误。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1891766|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

I got this error because the python script I was trying to submit was called pyspark.py (facepalm). The fix was to set my PYTHONPATH as recommended above, then rename the script to pyspark_test.py and clean up the pyspark.pyc that was created based on my scripts original name and that cleared this error up.

blocks|key|1891821|text|对于DSE+(DataStax+Cassandra+&+Spark)，需要将以下位置添加到PYTHONPATH|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1891822|export+PYTHONPATH=/usr/share/dse/resources/spark/python:$PYTHONPATH|code-block|syntax|javascript|1891823|然后使用dse+pyspark获取path中的模块。|1891824|dse+pyspark|1891825|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

In the case of DSE (DataStax Cassandra &amp; Spark) 
The following location needs to be added to PYTHONPATH

<pre><code>export PYTHONPATH=/usr/share/dse/resources/spark/python:$PYTHONPATH
</code></pre>

Then use the dse pyspark to get the modules in path.

<pre><code>dse pyspark
</code></pre>

blocks|key|1891897|text|我也遇到了同样的问题，我将在上面提出的解决方案中添加一件事。当在Mac上使用Homebrew安装Spark时，你需要纠正py4j路径地址，以便在路径中包含libexec+(记住将py4j版本更改为你已有的版本)；|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1891898|PYTHONPATH=$SPARK_HOME/libexec/python/lib/py4j-0.9-src.zip:$PYTHONPATH|code-block|syntax|javascript|1891899|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

I had this same problem and would add one thing to the proposed solutions above. When using Homebrew on Mac OS X to install Spark you will need to correct the py4j path address to include libexec in the path (remembering to change py4j version to the one you have);

<pre><code>PYTHONPATH=$SPARK_HOME/libexec/python/lib/py4j-0.9-src.zip:$PYTHONPATH
</code></pre>

blocks|key|1892082|text|您还可以创建一个Docker容器，将Alpine作为操作系统，并将Python和Pyspark安装为包。这将使其全部集装箱化。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1892083|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

You can also create a Docker container with Alpine as the OS and the install Python and Pyspark as packages. That will have it all containerised.

blocks|key|1678383|text|!pip+install+pyspark|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|1678384|在Jupyter+notebook或google+colab。不要忘记在colab+notebook顶部列出Restart+Runtime|unstyled|offset|length|style|CODE|1678385|entityMap^0|0|1I|F|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|N|8|@$G|O|H|P|I|J]]|9|@]|A|$]]|$1|K|3|-4|5|F|7|Q|8|@]|9|@]|A|$]]]|L|$]]

<pre><code>!pip install pyspark
</code></pre>
in Jupyter notebook or google colab. Do not forget to do <code>Restart Runtime</code> listed on top of colab notebook

This is a copy of someone else's question on another forum that was never answered, so I thought I'd re-ask it here, as I have the same issue. (See <a href="http://geekple.com/blogs/feeds/Xgzu7/posts/351703064084736">http://geekple.com/blogs/feeds/Xgzu7/posts/351703064084736</a>)

I have Spark installed properly on my machine and am able to run python programs with the pyspark modules without error when using ./bin/pyspark as my python interpreter.

However, when I attempt to run the regular Python shell, when I try to import pyspark modules I get this error:

<pre><code>from pyspark import SparkContext
</code></pre>

and it says

<pre><code>"No module named pyspark".
</code></pre>

How can I fix this? Is there an environment variable I need to set to point Python to the pyspark headers/libraries/etc.? If my spark installation is /spark/, which pyspark paths do I need to include? Or can pyspark programs only be run from the pyspark interpreter?

importing pyspark in python shell

Cassandra

解释器

容器

Python

CentOS

Spark 

这是另一个论坛上其他人的问题的副本，没有人回答，所以我想我应该在这里再问一次，因为我也有同样的问题。(参见)我已经在我的机器上正确安装了Spark，并且在使用./bin/pyspark作为我的python解释器时，能够正确地运行带有pyspark模块的python程序。然而，当我试图运行常规的Python shell时，当我试图导入pyspark模块时，我得到了这个错误：from pyspark 

问在python shell中导入pyspark
EN

回答 16

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在python shell中导入pysparkEN

回答 16

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在python shell中导入pyspark
EN