blocks|key|1879321|text|我不相信spark.driver.extraJavaOptions参数存在。对于spark.executor.extraJavaOptions，你似乎有一个拼写错误。试试这个：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1879322|--conf+"spark.executor.extraJavaOptions=-Dlog4j.configuration=/tmp/log4j.properties"|code-block|syntax|javascript|1879323|entityMap^0|4|T|14|V|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]|$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|S|8|@]|D|@]|E|$]]]|L|$]]

I don't believe the <code>spark.driver.extraJavaOptions</code> parameter exists. For <code>spark.executor.extraJavaOptions</code> it appears you have a typo. Try this:

<pre><code>--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=/tmp/log4j.properties"
</code></pre>

blocks|key|1879471|text|对于驱动程序/外壳程序，您可以在运行spark-shell或spark-submit脚本时使用--driver-java-options进行设置。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1879472|在Spark中，您不能设置--conf+spark.driver.extraJavaOptions，因为它是在启动JVM之后设置的。在使用spark提交脚本时，--driver-java-options会在启动运行驱动程序的JVM+(例如java+-Dblah+MyClass)时使用这些选项。|1879473|请注意，-Dlog4j.configuration属性应该是一个有效的URL，因此如果它来自文件系统中的某个位置，请使用file:+URL。如果资源变量无法转换为URL，例如由于MalformedURLException，那么log4j将从类路径中搜索资源。|1879474|例如，使用自定义的log4j.properties文件；|1879475|./spark-shell+--driver-java-options+"-Dlog4j.configuration=file:///etc/spark/my-conf/log4j.warnonly.properties"|code-block|syntax|javascript|1879476|entityMap^0|I|B|U|C|1B|L|0|D|10|28|L|3B|J|0|4|L|1O|5|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]|$9|V|A|W|B|C]|$9|X|A|Y|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|Z|8|@$9|10|A|11|B|C]|$9|12|A|13|B|C]|$9|14|A|15|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|16|8|@$9|17|A|18|B|C]|$9|19|A|1A|B|C]]|D|@]|E|$]]|$1|J|3|K|5|6|7|1B|8|@]|D|@]|E|$]]|$1|L|3|M|5|N|7|1C|8|@]|D|@]|E|$O|P]]|$1|Q|3|-4|5|6|7|1D|8|@]|D|@]|E|$]]]|R|$]]

For the driver/shell you can set this with the <code>--driver-java-options</code> when running <code>spark-shell</code> or <code>spark-submit</code> scripts. 

In Spark you cannot set <code>--conf spark.driver.extraJavaOptions</code> because that is set after the JVM is started. When using the spark submit scripts <code>--driver-java-options</code> substitutes these options into the launch of the JVM (e.g. <code>java -Dblah MyClass</code>) that runs the driver.

Note that the <code>-Dlog4j.configuration</code> property should be a valid URL, so if its from somewhere on your file system use <code>file:</code> URL. If the resource variable cannot be converted to a URL, for example due to a MalformedURLException, then log4j will search for the resource from the classpath.

For example, to use a custom log4j.properties file;

<pre><code>./spark-shell --driver-java-options "-Dlog4j.configuration=file:///etc/spark/my-conf/log4j.warnonly.properties"
</code></pre>

blocks|key|1879535|text|只有几个细节被忽略了。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1879536|conf标志应如下所示：|1879537|--conf+spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties"+--conf+spark.driver.extraJavaOptions="-Dlog4j.configuration=/tmp/log4j.properties"+--files+/tmp/log4j.properties|offset|length|style|CODE|1879538|您还需要使用--files参数将log4j.properties文件上传到集群，执行者可以在集群中访问该文件。此外，正如上面所述的配置，假设您使用的是client模式，在cluster中，两个配置将使用相同的相对路径：-Dlog4j.configuration=log4j.properties|1879539|备注:如果您的日志覆盖还需要额外的依赖项，您可能还需要提供它们：--conf+spark.driver.extraClassPath=custom-log4j-appender.jar请参阅：custom-log4j-appender-in-spark-executor|1879540|祝好运|1879541|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/32843186/custom-log4j-appender-in-spark-executor^0|0|0|0|5C|0|6|7|G|G|23|6|2D|7|31|12|0|W|1O|2O|13|0|0|0^^$0|@$1|2|3|4|5|6|7|X|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Y|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|Z|8|@$F|10|G|11|H|I]]|9|@]|A|$]]|$1|J|3|K|5|6|7|12|8|@$F|13|G|14|H|I]|$F|15|G|16|H|I]|$F|17|G|18|H|I]|$F|19|G|1A|H|I]|$F|1B|G|1C|H|I]]|9|@]|A|$]]|$1|L|3|M|5|6|7|1D|8|@$F|1E|G|1F|H|I]]|9|@$F|1G|G|1H|1|1I]]|A|$]]|$1|N|3|O|5|6|7|1J|8|@]|9|@]|A|$]]|$1|P|3|-4|5|6|7|1K|8|@]|9|@]|A|$]]]|Q|$R|$5|S|T|U|A|$V|W]]]]

Just a couple of details are off.

The conf flags should look like this: 
<code>--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties"
--conf spark.driver.extraJavaOptions="-Dlog4j.configuration=/tmp/log4j.properties"
--files /tmp/log4j.properties</code>

You'll also need to use the <code>--files</code> param to upload the <code>log4j.properties</code> file to the cluster, where executors can get to it. Also, as the configs are stated above assumes that you're using <code>client</code> mode, in <code>cluster</code> both configs would use the same relative path: <code>-Dlog4j.configuration=log4j.properties</code>

P.S. if your logging overrides also require additional dependencies you may need to provide them as well: <code>--conf spark.driver.extraClassPath=custom-log4j-appender.jar</code> See: <a href="https://stackoverflow.com/questions/32843186/custom-log4j-appender-in-spark-executor">custom-log4j-appender-in-spark-executor</a>

Good luck

I'm trying to override Spark's default <code>log4j.properties</code>, but haven't had any luck. I tried the adding the following to spark-submit:

<pre><code>--conf "spark.executor.extraJavaOptions=Dlog4j.configuration=/tmp/log4j.properties" 
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=/tmp/log4j.properties"
</code></pre>

But that didn't seem to work. I also tried using <code>--files</code> option in <code>spark-submit</code> and that also didn't seem to work. Has anyone got logging setup so you have a <code>log4j.properties</code> file per driver and not using the default?

I'm using Mesos and Marathon to run the Spark driver. I wasn't sure of the <code>--files</code> option and I couldn't find any examples of how it's used and what it does exactly.

I would also like to mention that I manually uploaded the <code>log4j.properties</code> file to all my nodes that had my changes for testing.

Version of Spark is 1.1.0 as of right now.

How to override Spark's log4j.properties per driver?

文件系统

Spark 

我正在尝试覆盖Spark的默认log4j.properties，但一直没有任何成功。我尝试在spark-submit中添加以下内容：--conf "spark.executor.extraJavaOptions=Dlog4j.configuration=/tmp/log4j.properties"  --conf "s...

问如何覆盖每个驱动程序的log4j.properties？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何覆盖每个驱动程序的log4j.properties？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何覆盖每个驱动程序的log4j.properties？
EN