首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >hive mapreduce reducer 调优

hive mapreduce reducer 调优

作者头像
大数据工程师-公子
发布2019-03-14 15:41:12
2.9K0
发布2019-03-14 15:41:12
举报

hive参数配置详情可查官方文档:Hive Configuration+Properties

本文针对 reducer 调优,主要涉及一下三个参数:

hive.exec.reducers.bytes.per.reducer

Default Value: 1,000,000,000 prior to Hive 0.14.0; 256 MB (256,000,000) in Hive 0.14.0 and later Added In: Hive 0.2.0; default changed in 0.14.0 with HIVE-7158 (and HIVE-7917) Size per reducer. The default in Hive 0.14.0 and earlier is 1 GB, that is, if the input size is 10 GB then 10 reducers will be used. In Hive 0.14.0 and later the default is 256 MB, that is, if the input size is 1 GB then 4 reducers will be used.

说明:每个reducer能够处理的文件大小。这个参数控制一个job会有多少个reducer来处理,依据的是输入文件的总大小。官方默认值:1G

查看配置的默认值:

hive> set hive.exec.reducers.bytes.per.reducer;
hive.exec.reducers.bytes.per.reducer=1024000000

临时调参:

hive> set hive.exec.reducers.bytes.per.reducer=15364000000;
hive>  set hive.exec.reducers.bytes.per.reducer;
hive.exec.reducers.bytes.per.reducer=15364000000

mapred.reduce.tasks

  • Default Value: -1
  • Added In: Hive 0.1.0 The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is “local”. Hadoop set this to 1 by default, whereas Hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers.

含义:设置每个job的reduce数。官方默认值为:-1

查看配置的默认值:

hive> set mapred.reduce.tasks;
mapred.reduce.tasks=-1

临时调参:

hive> set mapred.reduce.tasks=100;
hive> set mapred.reduce.tasks;
mapred.reduce.tasks=100

hive.exec.reducers.max

  • Default Value: 999 prior to Hive 0.14.0; 1009 in Hive 0.14.0 and later
  • Added In: Hive 0.2.0; default changed in 0.14.0 with HIVE-7158 (and HIVE-7917) Maximum number of reducers that will be used. If the one specified in the configuration property mapred.reduce.tasks is negative, Hive will use this as the maximum number of reducers when automatically determining the number of reducers.

含义:设置最大的reduce数。如果mapreduce计算的reduce数超过设定值的值,则取设置的值。

查看配置的默认值:

hive> set hive.exec.reducers.max;
hive.exec.reducers.max=1099

临时调参:

hive> set hive.exec.reducers.max=999;
hive> set hive.exec.reducers.max;
hive.exec.reducers.max=999

以上所有set的值,都只是临时调整,不会改变配置文件中的配置。当你重新打开hive会话,查看相应的配置就一目了然了。

本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
原始发表:2017年07月20日,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • hive.exec.reducers.bytes.per.reducer
  • mapred.reduce.tasks
  • hive.exec.reducers.max
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档