hive export报错找不到DistCpOptions类

原创

mikealzhou

发布于 2019-03-14 16:05:51

2.2K0

发布于 2019-03-14 16:05:51

文章被收录于专栏：大数据平台TBDS大数据平台TBDS

一、故障现象

执行hive的export命令：

export table dbname.customer_address to '/tmp/testuser/hiveexport/' ;

报错如下所示：

return code -101 from org.apache.hadoop.hive.ql.exec.copytask org/apache/hadoop/tools/distcpoptions

从上述报错，浏览社区没得到任何有价值的信息。通过查看hive的源码，也没有有效的信息。因此，我们需要查看hive的详细日志。

从hive的日志文件里面找到了以下内容：

报错如下（片段）：

2019-03-14 15:25:41 INFO common.FileUtils: Source is 106566465 bytes. (MAX: 33554432)
2019-03-14 15:25:41 INFO common.FileUtils: Launch distributed copy (distcp) job.
2019-03-14 15:25:41 ERROR metadata.Hive: Failed to move: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
2019-03-14 15:25:41 ERROR exec.Task: Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
        at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2895)
        at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:3205)
        at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1920)
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:364)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2084)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1755)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1463)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1181)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
        at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
        at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)

--------------------------------------------------------------------------

二、故障原因

hive 最后移动数据的时候，需要调用hadoop-distcp-X.X.X.jar,

解决方法：只需要把$HADOOP_HOME/share/hadoop/tools/lib/hadoop-distcp-x.x.x.jar 拷贝到 $HIVE_HOME/lib下面，重启hive即可。

三、进一步知识点

当然，如果hive表在HDFS上面存储的文件只要不超过33554432字节（也就是32M），就不需要使用到 org/apache/hadoop/tools/DistCpOptions 类，那么就不用把hadoop-distcp-x.x.x.jar 架包拷贝到hive的lib目录。这也是上面报错 “common.FileUtils: Source is 106566465 bytes. (MAX: 33554432)” 的原因所在。

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

hive