配置单元始终在本地模式下运行映射作业
配置单元始终在本地模式下运行映射作业
提问于 2014-04-30 00:02:44
我们正在使用Hive (0.13.0)测试多节点hadoop集群(2.4.0)。集群运行得很好，但是当我们在hive中运行查询时，映射的作业总是在本地执行。例如：
如果不使用hive-site.xml (实际上，除了默认配置文件之外没有任何配置文件)，我们将设置mapred.job.tracker：
hive> SET mapred.job.tracker=192.168.7.183:8032;并运行查询：
hive> select count(1) from suricata;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
OpenJDK 64-Bit Server VM warning: You have loaded library /hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/04/29 12:48:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
Execution log at: /tmp/hadoopuser/hadoopuser_20140429124747_badfcce6-620e-4718-8c3b-e4ef76bdba7e.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2014-04-29 12:48:05,450 null map = 0%, reduce = 0%
.......
.......
2014-04-29 12:52:26,982 null map = 100%, reduce = 100%
Ended Job = job_local1983771849_0001
Execution completed successfully
**MapredLocal task succeeded**
OK
266559841
Time taken: 270.176 seconds, Fetched: 1 row(s)我们遗漏了什么？
回答 2
Stack Overflow用户
发布于 2015-03-20 16:44:38
将hive.exec.mode.local.auto设置为false，这将禁用配置单元中的本地模式执行
Stack Overflow用户
发布于 2017-09-01 14:50:49
对于每个查询，编译器都会生成map-reduce作业的DAG。如果作业在本地模式下运行，请检查以下属性：
mapreduce.framework.name=local;
hive.exec.mode.local.auto=false;如果启用了自动选项，则配置单元将在以下情况下以本地模式运行作业
Total input size < hive.exec.mode.local.auto.inputbytes.max
Total number of map tasks < hive.exec.mode.local.auto.tasks.max
Total number of reduce tasks =< 1 or 0从0.7开始，可以使用这些选项
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/23369741
复制相关文章
相似问题
腾讯云开发者
