首页
学习
活动
专区
圈层
工具
发布
首页
学习
活动
专区
圈层
工具
MCP广场
社区首页 >问答首页 >在集群中启动MapReduce作业失败,退出代码为-1000,并且job.jar不存在

在集群中启动MapReduce作业失败,退出代码为-1000,并且job.jar不存在
EN

Stack Overflow用户
提问于 2018-08-25 17:05:40
回答 1查看 590关注 0票数 0

我试图在java代码中启动mapreduce作业,并将作业提交给纱线。但是得到了以下错误:

代码语言:javascript
运行
复制
2018-08-26 00:46:26,075 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-08-26 00:46:27,526 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at hdcluster01/10.211.55.22:8032
2018-08-26 00:46:28,135 WARN  [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2018-08-26 00:46:28,217 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(280)) - Total input paths to process : 1
2018-08-26 00:46:28,254 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:1
2018-08-26 00:46:28,364 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_1535213323614_0008
2018-08-26 00:46:28,484 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(204)) - Submitted application application_1535213323614_0008
2018-08-26 00:46:28,506 INFO  [main] mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://hdcluster01:8088/proxy/application_1535213323614_0008/
2018-08-26 00:46:28,506 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_1535213323614_0008
2018-08-26 00:46:32,536 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1355)) - Job job_1535213323614_0008 running in uber mode : false
2018-08-26 00:46:32,537 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1362)) -  map 0% reduce 0%
2018-08-26 00:46:32,547 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1375)) - Job job_1535213323614_0008 failed with state FAILED due to: Application application_1535213323614_0008 failed 2 times due to AM Container for appattempt_1535213323614_0008_000002 exited with  exitCode: -1000 due to: File file:/tmp/hadoop-yarn/staging/nasuf/.staging/job_1535213323614_0008/job.jar does not exist
.Failing this attempt.. Failing the application.
2018-08-26 00:46:32,570 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Counters: 0

错误:

代码语言:javascript
运行
复制
Job job_1535213323614_0008 failed with state FAILED due to: Application application_1535213323614_0008 failed 2 times due to AM Container for appattempt_1535213323614_0008_000002 exited with  exitCode: -1000 due to: File file:/tmp/hadoop-yarn/staging/nasuf/.staging/job_1535213323614_0008/job.jar does not exist
.Failing this attempt.. Failing the application.

我搞不懂我为什么会犯这个错误。我可以在命令行中成功地运行jar文件,但在java代码中失败。我检查了路径,路径/tmp/hadoop纱线/甚至不存在。本地用户是nasuf,运行hadoop的用户是平行的,而不是相同的。本地操作系统是MacOS,hadoop运行在Centos7中。

映射程序代码如下:

代码语言:javascript
运行
复制
public class WCMapper extends Mapper<LongWritable, Text, Text, LongWritable> {

    @Override
    protected void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException {

        String line = value.toString();
        String[] words = StringUtils.split(line, " ");

        for (String word: words) {
            context.write(new Text(word), new LongWritable(1));
        }

    }
}

和减速机代码如下:

代码语言:javascript
运行
复制
public class WCReducer extends Reducer<Text, LongWritable, Text, LongWritable>{

    @Override
    protected void reduce(Text key, Iterable<LongWritable> values, Context context) 
            throws IOException, InterruptedException {

        long count = 0;
        for (LongWritable value: values) {
            count += value.get();
        }

        context.write(key, new LongWritable(count));

    }

}

以及运行代码如下:

代码语言:javascript
运行
复制
public class WCRunner {

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        Configuration conf = new Configuration();
        conf.set("mapreduce.job.jar", "wc.jar");
        conf.set("mapreduce.framework.name", "yarn");
        conf.set("yarn.resourcemanager.hostname", "hdcluster01");
        conf.set("yarn.nodemanager.aux-services", "mapreduce_shuffle");
        Job job = Job.getInstance(conf);

        job.setJarByClass(WCRunner.class);

        job.setMapperClass(WCMapper.class);
        job.setReducerClass(WCReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(LongWritable.class);

        FileInputFormat.setInputPaths(job, new Path("hdfs://hdcluster01:9000/wc/srcdata"));

        FileOutputFormat.setOutputPath(job, new Path("hdfs://hdcluster01:9000/wc/output3"));

        job.waitForCompletion(true);

    }

}

有什么人能帮忙吗?非常感谢!

EN

回答 1

Stack Overflow用户

发布于 2018-08-26 03:31:08

我已经解决了这个问题。只需将core-site.xml放到类路径中,或者在代码中添加以下配置:

代码语言:javascript
运行
复制
conf.set("hadoop.tmp.dir", "/home/parallels/app/hadoop-2.4.1/data/");
票数 1
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/52019490

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档