我尝试按照这个http://hadoop.apache.org/common/docs/stable/single_node_setup.html文档安装Hadoop。当我尝试执行这段代码时
bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
我得到了以下异常
java.lang.OutOfMemoryError: Java heap space
请建议一个解决方案,以便我可以试用这个例子。下面列出了整个异常。我是Hadoop的新手,我可能做过一些愚蠢的事情。任何建议都将受到高度赞赏。
anuj@anuj-VPCEA13EN:~/hadoop$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
11/12/11 17:38:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library
11/12/11 17:38:22 INFO mapred.FileInputFormat: Total input paths to process : 7
11/12/11 17:38:22 INFO mapred.JobClient: Running job: job_local_0001
11/12/11 17:38:22 INFO util.ProcessTree: setsid exited with exit code 0
11/12/11 17:38:22 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@e49dcd
11/12/11 17:38:22 INFO mapred.MapTask: numReduceTasks: 1
11/12/11 17:38:22 INFO mapred.MapTask: io.sort.mb = 100
11/12/11 17:38:22 WARN mapred.LocalJobRunner: job_local_0001
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
11/12/11 17:38:23 INFO mapred.JobClient: map 0% reduce 0%
11/12/11 17:38:23 INFO mapred.JobClient: Job complete: job_local_0001
11/12/11 17:38:23 INFO mapred.JobClient: Counters: 0
11/12/11 17:38:23 INFO mapred.JobClient: Job Failed: NA
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1257)
at org.apache.hadoop.examples.Grep.run(Grep.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.Grep.main(Grep.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
发布于 2011-12-11 20:55:38
您可以通过编辑conf/mapred-site.xml文件并添加以下属性来分配更多内存:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024m</value>
</property>
这将使用更多的堆空间启动hadoop JVM。
发布于 2012-01-30 19:29:32
另一种可能是编辑包含export HADOOP_CLIENT_OPTS="-Xmx128m $HADOOP_CLIENT_OPTS"
的hadoop-env.sh
。把128m改成1024m对我很有帮助(Debian上的Hadoop 1.0.0.1 )。
发布于 2016-05-06 22:09:50
您需要对mapreduce.{map|reduce}.java.opts
和mapreduce.{map|reduce}.memory.mb
进行调整。
例如:
hadoop jar <jarName> <fqcn> \
-Dmapreduce.map.memory.mb=4096 \
-Dmapreduce.map.java.opts=-Xmx3686m
here是一个很好的资源,它回答了这个问题
https://stackoverflow.com/questions/8464048
复制相似问题