blocks|key|18240|text|我试图从使用spark-submit执行的python代码连接到mysql。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|18241|我使用的是使用Ambari的HDP沙箱。尝试了很多选项，如--jars、--driver-class-path等，但都不起作用。|18242|解决方案|18243|在/usr/local/miniconda/lib/python2.7/site-packages/pyspark/jars/中复制jar|18244|到目前为止，我不确定这是一个解决方案还是一个快速解决方案，但由于我在POC上工作，所以它对我来说是可行的。|18245|entityMap^0|6|C|0|T|6|10|J|0|0|1|1Q|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@$9|Q|A|R|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|S|8|@$9|T|A|U|B|C]|$9|V|A|W|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|X|8|@]|D|@]|E|$]]|$1|J|3|K|5|6|7|Y|8|@$9|Z|A|10|B|C]]|D|@]|E|$]]|$1|L|3|M|5|6|7|11|8|@]|D|@]|E|$]]|$1|N|3|-4|5|6|7|12|8|@]|D|@]|E|$]]]|O|$]]

I was trying to connect to mysql from the python code that was executed using <code>spark-submit</code>.
I was using HDP sandbox that was using Ambari. Tried lot of options such as <code>--jars</code>, <code>--driver-class-path</code>, etc, but none worked.
<h3>Solution</h3>
Copy the jar in <code>/usr/local/miniconda/lib/python2.7/site-packages/pyspark/jars/</code>
As of now I'm not sure if it's a solution or a quick hack, but since I'm working on POC so it kind of works for me.

blocks|key|18091|text|只需使用--jars参数。Spark将与executors共享这些jars+(以逗号分隔)。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|18092|entityMap^0|4|6|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|K|8|@]|D|@]|E|$]]]|G|$]]

Just use the <code>--jars</code> parameter. Spark will share those jars (comma-separated) with the executors.

blocks|key|4525868|text|为所有其他jars指定完整路径是可行的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4525869|./bin/spark-submit+--class+"SparkTest"+--master+local[*]+--jars+/fullpath/first.jar,/fullpath/second.jar+/fullpath/your-program.jar|code-block|syntax|javascript|4525870|或者通过添加如下行在conf/spark-defaults.conf中添加jars：|4525871|spark.driver.extraClassPath+/fullpath/firs.jar:/fullpath/second.jar
spark.executor.extraClassPath+/fullpath/firs.jar:/fullpath/second.jar|4525872|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Specifying full path for all additional jars works.

<pre><code>./bin/spark-submit --class "SparkTest" --master local[*] --jars /fullpath/first.jar,/fullpath/second.jar /fullpath/your-program.jar
</code></pre>

Or add jars in conf/spark-defaults.conf by adding lines like:

<pre><code>spark.driver.extraClassPath /fullpath/firs.jar:/fullpath/second.jar
spark.executor.extraClassPath /fullpath/firs.jar:/fullpath/second.jar
</code></pre>

blocks|key|4525923|text|在conf/spark-defaults.conf中添加时，您可以使用*将所有jars导入到一个文件夹中。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4525924|spark.driver.extraClassPath+/fullpath/*
spark.executor.extraClassPath+/fullpath/*|code-block|syntax|javascript|4525925|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You can use * for import all jars into a folder when adding in conf/spark-defaults.conf .

<pre><code>spark.driver.extraClassPath /fullpath/*
spark.executor.extraClassPath /fullpath/*
</code></pre>

blocks|key|4526004|text|在Spark+2.3中，您只需设置--jars选项。文件路径应该在方案前面加上，例如：file:////home/hadoop/spark/externaljsrs/*或file:////home/hadoop/spark/externaljars/abc.jar,file:////home/hadoop/spark/externaljars/def.jar：file:///<absolute+path+to+the+jars>|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4526005|entityMap^0|17|15|2D|2N|51|Z|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]|$9|M|A|N|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|O|8|@]|D|@]|E|$]]]|G|$]]

In Spark 2.3 you need to just set the --jars option. The file path should be prepended with the scheme though ie <code>file:///&lt;absolute path to the jars&gt;</code>
Eg : <code>file:////home/hadoop/spark/externaljsrs/*</code> or <code>file:////home/hadoop/spark/externaljars/abc.jar,file:////home/hadoop/spark/externaljars/def.jar</code>

blocks|key|18376|text|您可以使用--jars+$(echo+/Path/+to+/Your/+Jars+/*.jar+%7C+tr‘'，')来包含整个Jars文件夹。所以，spark-submit+--+class+com.yourClass+\+--jars+$(echo+/Path/To/Your/Jars/*.jar+%7C+tr‘'，')+\+...|type|unstyled|depth|inlineStyleRanges|entityRanges|data|18377|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

You can use --jars $(echo /Path/To/Your/Jars/*.jar | tr ' ' ',') to include entire folder of Jars.
So,
spark-submit -- class com.yourClass \
 --jars $(echo /Path/To/Your/Jars/*.jar | tr ' ' ',') \
 ...

blocks|key|18489|text|将包含以,分隔的jar文件路径的--jars传递给spark-submit。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|18490|供参考：|18491|--driver-class-path+is+used+to+mention+"extra"+jars+to+add+to+the+"driver"+of+the+spark+job

--driver-library-path+is+used+to+"change"+the+default+library+path+for+the+jars+needed+for+the+spark+driver

--driver-class-path+will+only+push+the+jars+to+the+driver+machine.+If+you+want+to+send+the+jars+to+"executors",+you+need+to+use+--jars|code-block|syntax|javascript|18492|要以编程方式设置jars，请设置以下配置:使用逗号分隔的jars列表设置spark.yarn.dist.jars。|18493|例如：|18494|from+pyspark.sql+import+SparkSession

spark+=+SparkSession+\
++++++++.builder+\
++++++++.appName("Spark+config+example")+\
++++++++.config("spark.yarn.dist.jars",+"<path-to-jar/test1.jar>,<path-to-jar/test2.jar>")+\
++++++++.getOrCreate()|18495|entityMap^0|4|1|8|3|G|6|P|C|0|0|0|10|K|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@$9|V|A|W|B|C]|$9|X|A|Y|B|C]|$9|Z|A|10|B|C]|$9|11|A|12|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|13|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|14|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|15|8|@$9|16|A|17|B|C]]|D|@]|E|$]]|$1|O|3|P|5|6|7|18|8|@]|D|@]|E|$]]|$1|Q|3|R|5|J|7|19|8|@]|D|@]|E|$K|L]]|$1|S|3|-4|5|6|7|1A|8|@]|D|@]|E|$]]]|T|$]]

Pass <code>--jars</code> with the path of <code>jar</code> files separated by <code>,</code> to <code>spark-submit</code>.

For reference:

<pre><code>--driver-class-path is used to mention "extra" jars to add to the "driver" of the spark job

--driver-library-path is used to "change" the default library path for the jars needed for the spark driver

--driver-class-path will only push the jars to the driver machine. If you want to send the jars to "executors", you need to use --jars
</code></pre>

And to set the jars programatically set the following config:
<code>spark.yarn.dist.jars</code> with comma-separated list of jars.

Eg:

<pre><code>from pyspark.sql import SparkSession

spark = SparkSession \
 .builder \
 .appName("Spark config example") \
 .config("spark.yarn.dist.jars", "&lt;path-to-jar/test1.jar&gt;,&lt;path-to-jar/test2.jar&gt;") \
 .getOrCreate()
</code></pre>

blocks|key|4526017|text|对于--driver-class-path选项，您可以使用:作为分隔符来传递多个jars。下面是使用spark-shell命令的示例，但我猜同样的方法也适用于spark-submit|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4526018|++++spark-shell+--driver-class-path+/path/to/example.jar:/path/to/another.jar|code-block|syntax|javascript|4526019|Spark版本:+2.2.0|4526020|entityMap^0|2|J|T|1|1E|B|27|C|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]|$9|R|A|S|B|C]|$9|T|A|U|B|C]|$9|V|A|W|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|X|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|Y|8|@]|D|@]|E|$]]|$1|M|3|-4|5|6|7|Z|8|@]|D|@]|E|$]]]|N|$]]

For <code>--driver-class-path</code> option you can use <code>:</code> as delimeter to pass multiple jars.
Below is the example with <code>spark-shell</code> command but I guess the same should work with <code>spark-submit</code> as well

<pre><code> spark-shell --driver-class-path /path/to/example.jar:/path/to/another.jar
</code></pre>

Spark version: 2.2.0

blocks|key|18416|text|如果您使用的是属性文件，则可以在此处添加以下行：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|18417|spark.jars=jars/your_jar1.jar,...|code-block|syntax|javascript|18418|假设|18419|<your+root+from+where+you+run+spark-submit>
++%7C
++%7C-jars
++++++%7C-your_jar1.jar|18420|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

if you are using properties file you can add following line there:

<pre><code>spark.jars=jars/your_jar1.jar,...
</code></pre>

assuming that

<pre><code>&lt;your root from where you run spark-submit&gt;
 |
 |-jars
 |-your_jar1.jar
</code></pre>

I am trying to run a spark program where i have multiple jar files, if I had only one jar I am not able run. I want to add both the jar files which are in same location. I have tried the below but it shows a dependency error

<pre><code>spark-submit \
 --class "max" maxjar.jar Book1.csv test \
 --driver-class-path /usr/lib/spark/assembly/lib/hive-common-0.13.1-cdh​5.3.0.jar
</code></pre>

How can i add another jar file which is in the same directory?

I want add <code>/usr/lib/spark/assembly/lib/hive-serde.jar</code>.

spark submit add multiple jars in classpath

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我正在尝试运行一个spark程序，其中我有多个jar文件，如果我只有一个jar文件，我就无法运行。我想添加位于相同位置的两个jar文件。我已经尝试了下面的代码，但它显示了一个依赖错误spark-submit \  --class "max" maxjar.jar Book1.csv test \  --driver-class-path /usr/lib/spark/assembly/lib/hi

问spark提交在类路径中添加多个jars
EN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问spark提交在类路径中添加多个jarsEN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问spark提交在类路径中添加多个jars
EN