blocks|key|2574945|text|选项1+:|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|2574946|Glue使用spark上下文，您也可以将hadoop配置设置为aws+glue。因为内部动态框架是一种数据帧。|2574947|sc._jsc.hadoopConfiguration().set("mykey","myvalue")|code-block|syntax|javascript|2574948|我认为您还需要像这样添加相应的类|2574949|sc._jsc.hadoopConfiguration().set("mapred.output.committer.class",+"org.apache.hadoop.mapred.FileOutputCommitter")|2574950|2574951|+sc+=+SparkContext()

++++sc._jsc.hadoopConfiguration().set("mapreduce.fileoutputcommitter.algorithm.version","2")

++++glueContext+=+GlueContext(sc)
++++spark+=+glueContext.spark_session|2574952|2574953|在python中调试：|2574954|sc._conf.getAll()+//+print+this|2574955|2574956|sc.getConf.getAll.foreach(println)|2574957|选项2:|2574958|另一边你试着使用胶水的作业参数：|2574959|https://docs.aws.amazon.com/glue/latest/dg/add-job.html，它具有文档中提到的键值属性|2574960|'--myKey'+:+'value-for-myKey'++|2574961|您可以按照下面的屏幕截图编辑作业并使用--conf指定参数|CODE|2574962|​|2574963|📷|atomic|2574964|2574965|选项3:|2574966|如果您正在使用aws+cli，您可以在下面尝试...https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html|2574967|有趣的是，他们在文档中提到，不要像下面这样设置消息。|2574968|2574969|2574970|2574971|2574972|总结:我个人更喜欢option1，因为你有编程控制。|blockquote|2574973|2574974|entityMap|0|LINK|mutability|MUTABLE|url|https://docs.aws.amazon.com/glue/latest/dg/add-job.html|1|IMAGE|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/dc7c5f63c89a0dff36626da0f5299c19.png|imageAlt|2|https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html|3|https://ask.qcloudimg.com/http-save/yehe-900000/05d59d63862b77c100aeb5aefbeb2f46.png^0|0|5|0|0|0|0|0|0|0|0|0|0|0|0|0|4|0|0|0|1J|0|0|0|J|6|0|0|0|1|1|0|0|0|4|0|Q|2F|2|0|0|0|0|1|3|0|0|0|9|7|0|0^^$0|@$1|2|3|4|5|6|7|2E|8|@$9|2F|A|2G|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|2H|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|2I|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|2J|8|@]|D|@]|E|$]]|$1|O|3|P|5|J|7|2K|8|@]|D|@]|E|$K|L]]|$1|Q|3|-4|5|6|7|2L|8|@]|D|@]|E|$]]|$1|R|3|S|5|J|7|2M|8|@]|D|@]|E|$K|L]]|$1|T|3|-4|5|6|7|2N|8|@]|D|@]|E|$]]|$1|U|3|V|5|6|7|2O|8|@]|D|@]|E|$]]|$1|W|3|X|5|J|7|2P|8|@]|D|@]|E|$K|L]]|$1|Y|3|-4|5|6|7|2Q|8|@]|D|@]|E|$]]|$1|Z|3|10|5|J|7|2R|8|@]|D|@]|E|$K|L]]|$1|11|3|12|5|6|7|2S|8|@$9|2T|A|2U|B|C]]|D|@]|E|$]]|$1|13|3|14|5|6|7|2V|8|@]|D|@]|E|$]]|$1|15|3|16|5|6|7|2W|8|@]|D|@$9|2X|A|2Y|1|2Z]]|E|$]]|$1|17|3|18|5|J|7|30|8|@]|D|@]|E|$K|L]]|$1|19|3|1A|5|6|7|31|8|@$9|32|A|33|B|1B]]|D|@]|E|$]]|$1|1C|3|1D|5|6|7|34|8|@]|D|@]|E|$]]|$1|1E|3|1F|5|1G|7|35|8|@]|D|@$9|36|A|37|1|38]]|E|$]]|$1|1H|3|1D|5|6|7|39|8|@]|D|@]|E|$]]|$1|1I|3|1J|5|6|7|3A|8|@$9|3B|A|3C|B|C]]|D|@]|E|$]]|$1|1K|3|1L|5|6|7|3D|8|@]|D|@$9|3E|A|3F|1|3G]]|E|$]]|$1|1M|3|1N|5|6|7|3H|8|@]|D|@]|E|$]]|$1|1O|3|1D|5|6|7|3I|8|@]|D|@]|E|$]]|$1|1P|3|1F|5|1G|7|3J|8|@]|D|@$9|3K|A|3L|1|3M]]|E|$]]|$1|1Q|3|1D|5|6|7|3N|8|@]|D|@]|E|$]]|$1|1R|3|-4|5|6|7|3O|8|@]|D|@]|E|$]]|$1|1S|3|1T|5|1U|7|3P|8|@$9|3Q|A|3R|B|C]]|D|@]|E|$]]|$1|1V|3|-4|5|6|7|3S|8|@]|D|@]|E|$]]|$1|1W|3|-4|5|6|7|3T|8|@]|D|@]|E|$]]]|1X|$1Y|$5|1Z|20|21|E|$22|23]]|24|$5|25|20|26|E|$27|28|29|-4]]|2A|$5|1Z|20|21|E|$22|2B]]|2C|$5|25|20|26|E|$27|2D|29|-4]]]]

Option 1 :

Glue uses spark context you can set hadoop configuration to aws glue as well. since internally dynamic frame is kind of dataframe.

<pre><code>sc._jsc.hadoopConfiguration().set("mykey","myvalue")
</code></pre>

I think you neeed to add the correspodning class also like this 

<pre><code>sc._jsc.hadoopConfiguration().set("mapred.output.committer.class", "org.apache.hadoop.mapred.FileOutputCommitter")
</code></pre>

example snippet :

<pre><code> sc = SparkContext()

 sc._jsc.hadoopConfiguration().set("mapreduce.fileoutputcommitter.algorithm.version","2")

 glueContext = GlueContext(sc)
 spark = glueContext.spark_session
</code></pre>

To prove that that configuration exists ....

Debug in python :

<pre><code>sc._conf.getAll() // print this
</code></pre>

Debug in scala : 

<pre><code>sc.getConf.getAll.foreach(println)
</code></pre>

Option 2: 

Other side you try using job parameters of the glue :

<a href="https://docs.aws.amazon.com/glue/latest/dg/add-job.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/glue/latest/dg/add-job.html</a>
which has key value properties like mentioned in docs

<pre><code>'--myKey' : 'value-for-myKey' 
</code></pre>

you can follow below screen shot for editing job and specifying the parameters with <code>--conf</code>

<a href="https://i.stack.imgur.com/N7D42.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/N7D42.png" alt="enter image description here"></a>

Option 3: 
If you are using, aws cli you can try below...
<a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html</a>

Fun is they mentioned in the docs dont set message like below. but i dont know why it was exposed.

<a href="https://i.stack.imgur.com/463Ty.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/463Ty.png" alt="enter image description here"></a>

<blockquote>
 To sum up : I personally prefer option1 since you have
 programmatic control.
</blockquote>

blocks|key|2575031|text|进入胶水作业控制台，按如下所示编辑您的作业：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2575032|2575033|+Glue>作业>编辑Job>脚本库和作业参数(可选)>作业参数|blockquote|2575034|2575035|2575036|设置以下内容：|2575037|2575038|密钥：--conf值：|2575039|2575040|spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2|2575041|entityMap^0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|R|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|S|8|@]|9|@]|A|$]]|$1|C|3|D|5|E|7|T|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|U|8|@]|9|@]|A|$]]|$1|G|3|-4|5|6|7|V|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|W|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|X|8|@]|9|@]|A|$]]|$1|K|3|L|5|E|7|Y|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|N|3|O|5|6|7|10|8|@]|9|@]|A|$]]|$1|P|3|-4|5|6|7|11|8|@]|9|@]|A|$]]]|Q|$]]

Go to glue job console and edit your job as follows :
<blockquote>
Glue&gt; Jobs &gt; Edit your Job&gt; Script libraries and job parameters
(optional) &gt; Job parameters
</blockquote>
Set the following:
<blockquote>
key: --conf value:
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2
</blockquote>

I haven't been able to figure this out, but I'm trying to use a direct output committer with AWS Glue:

<code>spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2</code>

Is it possible to use this configuration with AWS Glue?

Use Spark fileoutputcommitter.algorithm.version=2 with AWS Glue

我还没有弄清楚这一点，但我正在尝试使用AWS Glue的直接输出提交者：spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2是否可以将此配置与AWS Glue一起使用？

问将Spark fileoutputcommitter.algorithm.version=2与AWS胶水配合使用
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问将Spark fileoutputcommitter.algorithm.version=2与AWS胶水配合使用EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问将Spark fileoutputcommitter.algorithm.version=2与AWS胶水配合使用
EN