blocks|key|4483025|text|简而言之，有两种方法可以使用pip安装包，具体取决于平台。首先，你需要安装任何你需要的东西，然后你就可以运行你的Spark步骤了。最简单的方法是使用emr-4.0.0和'command-runner.jar'：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4483026|from+boto.emr.step+import+JarStep
>>>+pip_step=JarStep(name="Command+Runner",
...+++++++++++++jar="command-runner.jar",
...+++++++++++++action_on_failure="CONTINUE",
...+++++++++++++step_args=['sudo','pip','install','arrow']
...+)
>>>+spark_step=JarStep(name="Spark+with+Command+Runner",
...++++++++++++++++++++jar="command-runner.jar",
...++++++++++++++++++++step_args=["spark-submit","/usr/lib/spark/examples/src/main/python/pi.py"]
...++++++++++++++++++++action_on_failure="CONTINUE"
)
>>>+step_list=conn.add_jobflow_steps(emr.jobflowid,+[pip_step,spark_step])|code-block|syntax|javascript|4483027|在2.x和3.x上，您以类似的方式使用script-runner.jar，只是您必须为scriptrunner指定完整的URI。|offset|length|4483028|编辑:对不起，我不知道你想通过控制台来做这件事。您也可以在控制台中添加相同的步骤。第一步是使用与上面相同的参数创建一个客户JAR。第二步是火花步。希望这能有所帮助！|4483029|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-script.html^0|0|0|17|C|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|V|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|W|8|@]|9|@$I|X|J|Y|1|Z]]|A|$]]|$1|K|3|L|5|6|7|10|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|11|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|T]]]]

In short, there are two ways to install packages with pip, depending on the platform. First, you install whatever you need and then you can run your Spark step. The easiest is to use emr-4.0.0 and 'command-runner.jar':

<pre><code>from boto.emr.step import JarStep
&gt;&gt;&gt; pip_step=JarStep(name="Command Runner",
... jar="command-runner.jar",
... action_on_failure="CONTINUE",
... step_args=['sudo','pip','install','arrow']
... )
&gt;&gt;&gt; spark_step=JarStep(name="Spark with Command Runner",
... jar="command-runner.jar",
... step_args=["spark-submit","/usr/lib/spark/examples/src/main/python/pi.py"]
... action_on_failure="CONTINUE"
)
&gt;&gt;&gt; step_list=conn.add_jobflow_steps(emr.jobflowid, [pip_step,spark_step])
</code></pre>

On 2.x and 3.x, you use script-runner.jar in a similar fashion except that you have to specify the full URI for <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-script.html" rel="noreferrer">scriptrunner</a>. 

EDIT:
Sorry, I didn't see that you wanted to do this through console. You can add the same steps in the console as well. The first step would be a Customer JAR with the same args as above. The second step is a spark step. Hope this helps!

blocks|key|4484099|text|这篇文章让我走上了正确的道路，但最终我选择了一个不同的解决方案。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4484100|boostrap.sh|offset|length|style|CODE|4484101|#!/bin/bash

sudo+python3+-m+pip+install+\
++++botocore+\
++++boto3+\
++++ujson+\
++++warcio+\
++++beautifulsoup4++\
++++lxml|code-block|syntax|javascript|4484102|create_emr_cluster.sh|4484103|#!/bin/bash

pem_file="~/.ssh/<your+pem+file>.pem"
bootstrap_path="s3://<path+without+filename>/"
subnet_id="subnet-<uniuqe+subnet+id>"
logs_path="s3://<log+directory+(optional)>/elasticmapreduce/"

aws+s3+cp+./bootstrap.sh+$bootstrap_path

ID=$(aws+emr+create-cluster+\
--name+spark-data-processing+\
--use-default-roles+\
--release-label+emr-5.30.1+\
--instance-count+2+\
--application+Name=Spark+Name=Hive+Name=Ganglia+Name=Zeppelin+\
--ec2-attributes+KeyName=<your+pem+file>,SubnetId=$subnet_id+\
--instance-type+m4.large+\
--bootstrap-actions+Path=${bootstrap_path}bootstrap.sh+\
--query+ClusterId+\
--output+text+\
--log-uri+${logs_path})|4484104|credit+to感谢他的帮助|4484105|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/EricRops^0|0|0|B|0|0|0|L|0|0|0|9|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|11|8|@$D|12|E|13|F|G]]|9|@]|A|$]]|$1|H|3|I|5|J|7|14|8|@]|9|@]|A|$K|L]]|$1|M|3|N|5|6|7|15|8|@$D|16|E|17|F|G]]|9|@]|A|$]]|$1|O|3|P|5|J|7|18|8|@]|9|@]|A|$K|L]]|$1|Q|3|R|5|6|7|19|8|@]|9|@$D|1A|E|1B|1|1C]]|A|$]]|$1|S|3|-4|5|6|7|1D|8|@]|9|@]|A|$]]]|T|$U|$5|V|W|X|A|$Y|Z]]]]

This post got me started down the right path but ultimately I ended on a different solution.
<code>boostrap.sh</code>
<pre class="lang-sh prettyprint-override"><code>#!/bin/bash

sudo python3 -m pip install \
 botocore \
 boto3 \
 ujson \
 warcio \
 beautifulsoup4 \
 lxml
</code></pre>
<code>create_emr_cluster.sh</code>
<pre class="lang-sh prettyprint-override"><code>#!/bin/bash

pem_file=&quot;~/.ssh/&lt;your pem file&gt;.pem&quot;
bootstrap_path=&quot;s3://&lt;path without filename&gt;/&quot;
subnet_id=&quot;subnet-&lt;uniuqe subnet id&gt;&quot;
logs_path=&quot;s3://&lt;log directory (optional)&gt;/elasticmapreduce/&quot;

aws s3 cp ./bootstrap.sh $bootstrap_path

ID=$(aws emr create-cluster \
--name spark-data-processing \
--use-default-roles \
--release-label emr-5.30.1 \
--instance-count 2 \
--application Name=Spark Name=Hive Name=Ganglia Name=Zeppelin \
--ec2-attributes KeyName=&lt;your pem file&gt;,SubnetId=$subnet_id \
--instance-type m4.large \
--bootstrap-actions Path=${bootstrap_path}bootstrap.sh \
--query ClusterId \
--output text \
--log-uri ${logs_path})
</code></pre>
<a href="https://github.com/EricRops" rel="nofollow noreferrer">credit to</a> for his help

blocks|key|4483132|text|对于Python+3-创建一个bash脚本：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4483133|#!/bin/bash+-xe
sudo+pip3+install+your_package|code-block|syntax|javascript|4483134|将其复制到S3，并从控制台设置引导程序操作以指向您的脚本。|4483135|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

For Python 3 - create a bash script:
<pre><code>#!/bin/bash -xe
sudo pip3 install your_package
</code></pre>
Copy it to S3, and set a bootstrap action from the console to point to your script.

I want to do something really basic, simply fire up a Spark cluster through the EMR console and run a Spark script that depends on a Python package (for example, <a href="https://github.com/crsmithdev/arrow">Arrow</a>). What is the most straightforward way of doing this?

How to bootstrap installation of Python modules on Amazon EMR?

Python

Spark 

Hive

我想做一些非常基本的事情，只需通过EMR控制台启动一个Spark集群，然后运行一个依赖于Python包的Spark脚本(例如，)。做这件事最直接的方法是什么？

问如何在Amazon EMR上引导Python模块的安装？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Amazon EMR上引导Python模块的安装？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Amazon EMR上引导Python模块的安装？
EN