blocks|key|4224460|text|为了保持网格上的所有内容，使用hadoop+streaming并使用单个reducer和cat作为映射器和reducer+(基本上是noop)+-使用MR标志添加压缩。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4224461|hadoop+jar+\
++++$HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar+\ 
++++-Dmapred.reduce.tasks=1+\
++++-Dmapred.job.queue.name=$QUEUE+\
++++-input+"$INPUT"+\
++++-output+"$OUTPUT"+\
++++-mapper+cat+\
++++-reducer+cat|code-block|syntax|javascript|4224462|如果您想要压缩添加|4224463|-Dmapred.output.compress=true+\+-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec|4224464|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

In order to keep everything on the grid use hadoop streaming with a single reducer and cat as the mapper and reducer (basically a noop) - add compression using MR flags.

<pre><code>hadoop jar \
 $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar \&lt;br&gt;
 -Dmapred.reduce.tasks=1 \
 -Dmapred.job.queue.name=$QUEUE \
 -input "$INPUT" \
 -output "$OUTPUT" \
 -mapper cat \
 -reducer cat
</code></pre>

If you want compression add 
 -Dmapred.output.compress=true \
-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec

blocks|key|2113283|text|hadoop+fs+-getmerge+<dir_of_input_files>+<mergedsinglefile>|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|2113284|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>hadoop fs -getmerge &lt;dir_of_input_files&gt; &lt;mergedsinglefile&gt;
</code></pre>

blocks|key|5642529|text|好的……我想出了一种使用hadoop+fs命令的方法-|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|5642530|hadoop+fs+-cat+[dir]/*+%7C+hadoop+fs+-put+-+[destination+file]|code-block|syntax|javascript|5642531|当我测试一个人能想到的it...any陷阱时，它起作用了吗？|5642532|谢谢!|5642533|entityMap^0|C|9|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@$9|R|A|S|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|T|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|U|8|@]|D|@]|E|$]]|$1|M|3|N|5|6|7|V|8|@]|D|@]|E|$]]|$1|O|3|-4|5|6|7|W|8|@]|D|@]|E|$]]]|P|$]]

okay...I figured out a way using <code>hadoop fs</code> commands - 

<pre><code>hadoop fs -cat [dir]/* | hadoop fs -put - [destination file]
</code></pre>

It worked when I tested it...any pitfalls one can think of?

Thanks!

blocks|key|4224405|text|如果您设置fuse将HDFS挂载到本地目录，那么您的输出可以是挂载的文件系统。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4224406|例如，我在本地将我们的HDFS挂载到/mnt/hdfs。我运行以下命令，它工作得很好：|offset|length|style|CODE|4224407|hadoop+fs+-getmerge+/reports/some_output+/mnt/hdfs/reports/some_output.txt|code-block|syntax|javascript|4224408|当然，使用fuse将HDFS挂载到本地目录还有其他原因，但这对我们来说是一个很好的副作用。|4224409|entityMap^0|0|I|9|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@$D|S|E|T|F|G]]|9|@]|A|$]]|$1|H|3|I|5|J|7|U|8|@]|9|@]|A|$K|L]]|$1|M|3|N|5|6|7|V|8|@]|9|@]|A|$]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

If you set up fuse to mount your HDFS to a local directory, then your output can be the mounted filesystem. 

For example, I have our HDFS mounted to <code>/mnt/hdfs</code> locally. I run the following command and it works great:

<pre><code>hadoop fs -getmerge /reports/some_output /mnt/hdfs/reports/some_output.txt
</code></pre>

Of course, there are other reasons to use fuse to mount HDFS to a local directory, but this was a nice side effect for us.

blocks|key|4224385|text|您可以使用HDFS0.21中新增的工具HDFSConcat来执行此操作，而不会产生复制成本。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4224386|entityMap|0|LINK|mutability|MUTABLE|url|http://hadoop.apache.org/hdfs/docs/r0.21.0/api/org/apache/hadoop/hdfs/tools/HDFSConcat.html^0|J|A|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

You can use the tool <a href="http://hadoop.apache.org/hdfs/docs/r0.21.0/api/org/apache/hadoop/hdfs/tools/HDFSConcat.html" rel="nofollow">HDFSConcat</a>, new in HDFS 0.21, to perform this operation without incurring the cost of a copy.

blocks|key|5642654|text|如果您在Hortonworks集群中工作，并且想要将HDFS位置中存在的多个文件合并为一个文件，那么您可以运行‘hadoop-streaming+2.7.1.2.3.2.0-2950.jar’jar，它将运行单个reducer并将合并后的文件放入HDFS输出位置。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5642655|$+hadoop+jar+/usr/hdp/2.3.2.0-2950/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.2.0-2950.jar+\|5642656|-Dmapred.reduce.tasks=1+\
-input+"/hdfs/input/dir"+\
-output+"/hdfs/output/dir"+\
-mapper+cat+\
-reducer+cat|code-block|syntax|javascript|5642657|您可以从Get+hadoop+streaming+jar下载此jar|offset|length|5642658|如果您正在编写spark作业，并且希望获得一个合并文件以避免多次RDD创建和性能瓶颈，请在转换RDD之前使用这段代码|5642659|sc.textFile("hdfs://...../part*).coalesce(1).saveAsTextFile("hdfs://...../filename)|style|CODE|5642660|这会将所有零件文件合并为一个文件，并将其再次保存到hdfs位置|5642661|entityMap|0|LINK|mutability|MUTABLE|url|https://javalibs.com/artifact/org.apache.hadoop/hadoop-streaming^0|0|0|0|4|O|0|0|0|0|2B|0|0^^$0|@$1|2|3|4|5|6|7|12|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|13|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|14|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|15|8|@]|9|@$K|16|L|17|1|18]]|A|$]]|$1|M|3|N|5|6|7|19|8|@]|9|@]|A|$]]|$1|O|3|P|5|6|7|1A|8|@$K|1B|L|1C|Q|R]]|9|@]|A|$]]|$1|S|3|T|5|6|7|1D|8|@]|9|@]|A|$]]|$1|U|3|-4|5|6|7|1E|8|@]|9|@]|A|$]]]|V|$W|$5|X|Y|Z|A|$10|11]]]]

If you are working in Hortonworks cluster and want to merge multiple file present in HDFS location into a single file then you can run 'hadoop-streaming-2.7.1.2.3.2.0-2950.jar' jar which runs single reducer and get the merged file into HDFS output location.

$ hadoop jar /usr/hdp/2.3.2.0-2950/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.2.0-2950.jar \

<pre><code>-Dmapred.reduce.tasks=1 \
-input "/hdfs/input/dir" \
-output "/hdfs/output/dir" \
-mapper cat \
-reducer cat
</code></pre>

You can download this jar from 
<a href="https://javalibs.com/artifact/org.apache.hadoop/hadoop-streaming" rel="nofollow noreferrer">Get hadoop streaming jar</a>

If you are writing spark jobs and want to get a merged file to avoid multiple RDD creations and performance bottlenecks use this piece of code before transforming your RDD

<code>sc.textFile("hdfs://...../part*).coalesce(1).saveAsTextFile("hdfs://...../filename)
</code>

This will merge all part files into one and save it again into hdfs location

blocks|key|2113358|text|所有的解决方案都等同于做一个|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2113359|hadoop+fs+-cat+[dir]/*+>+tmp_local_file++
hadoop+fs+-copyFromLocal+tmp_local_file+|code-block|syntax|javascript|2113360|这只意味着本地m/c+I/O处于数据传输的关键路径上。|2113361|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

All the solutions are equivalent to doing a 

<pre><code>hadoop fs -cat [dir]/* &gt; tmp_local_file 
hadoop fs -copyFromLocal tmp_local_file 
</code></pre>

it only means that the local m/c I/O is on the critical path of data transfer.

blocks|key|1674338|text|从Apache+Pig的角度解决这个问题，|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1674339|要通过Pig合并具有相同模式的两个文件，可以使用UNION命令|1674340|+A+=+load+'tmp/file1'+Using+PigStorage('\t')+as+....(schema1)
+B+=+load+'tmp/file2'+Using+PigStorage('\t')+as+....(schema1)+
+C+=+UNION+A,B
+store+C+into+'tmp/fileoutput'+Using+PigStorage('\t')|code-block|syntax|javascript|1674341|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

Addressing this from Apache Pig perspective,

To merge two files with identical schema via Pig, UNION command can be used

<pre><code> A = load 'tmp/file1' Using PigStorage('\t') as ....(schema1)
 B = load 'tmp/file2' Using PigStorage('\t') as ....(schema1) 
 C = UNION A,B
 store C into 'tmp/fileoutput' Using PigStorage('\t')
</code></pre>

I get multiple small files into my input directory which I want to merge into a single file without using the local file system or writing mapreds. Is there a way I could do it using hadoof fs commands or Pig?

Thanks!

Merging multiple files into one within Hadoop

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋 

腾讯云代码助手

CODING DevOps

Cloud Studio

SDK中心

API中心

命令行工具

我将多个小文件放到我的输入目录中，我希望将它们合并到一个文件中，而不使用本地文件系统或编写mapred。有没有办法可以用hadoof fs命令或者Pig来做呢？谢谢!

问在Hadoop中将多个文件合并为一个文件
EN

回答 8

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Hadoop中将多个文件合并为一个文件EN

回答 8

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Hadoop中将多个文件合并为一个文件
EN