WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-

别先生

发布于 2018-05-28 16:56:29

1K0

发布于 2018-05-28 16:56:29

文章被收录于专栏：别先生别先生

1、执行脚本程序报如下所示的错误：

 1 [hadoop@slaver1 script_hadoop]$  hadoop jar web_click_mr_hive.jar com.bie.hive.mr.ClickStreamThree /home/hadoop/data_hadoop/weblog/preprocess/output/2018-05-19 /home/hadoop/data_hadoop/weblog/preprocess/click_pv_out/2018-05-19
 2 18/05/20 17:09:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 3 Exception in thread "main" java.io.IOException: No FileSystem for scheme: C
 4     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
 5     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
 6     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
 7     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
 8     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
 9     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
10     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
11     at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:498)
12     at com.bie.hive.mr.ClickStreamThree.main(ClickStreamThree.java:207)
13     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
14     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
15     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16     at java.lang.reflect.Method.invoke(Method.java:606)
17     at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
18 [hadoop@slaver1 script_hadoop]$  hadoop jar web_click_mr_hive.jar com.bie.hive.mr.ClickStreamThree hdfs://slaver1:9000/home/hadoop/data_hadoop/weblog/preprocess/output/2018-05-19 hdfs://slaver1:9000/home/hadoop/data_hadoop/weblog/preprocess/click_pv_out/2018-05-19
19 18/05/20 17:10:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20 Exception in thread "main" java.io.IOException: No FileSystem for scheme: C
21     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
22     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
23     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
24     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
25     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
26     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
27     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
28     at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:498)
29     at com.bie.hive.mr.ClickStreamThree.main(ClickStreamThree.java:207)
30     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
31     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
32     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
33     at java.lang.reflect.Method.invoke(Method.java:606)
34     at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
35 [hadoop@slaver1 script_hadoop]$

2、猛一看，WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable，百度一搜这个错误，啥编译32/64位的问题，也有说改配置文件的问题，也许有的可能是这样改就解决问题了，而我想说，出现问题，看看报错原因，也许就可以很快的解决问题了。

然后呢，看下代码，是如下所示导致的错误，将下面的注释了，代码换成上面的就可以使用脚本程序执行代码了：

FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

//FileInputFormat.setInputPaths(job, new Path("C:/weblog/output"));
//FileOutputFormat.setOutputPath(job, new Path("c:/weblog/pageviews"));

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2018-05-20 ，如有侵权请联系 cloudcommunity@tencent.com 删除

nat