前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >YARN 的配置

YARN 的配置

作者头像
北漂的我
发布2019-05-29 10:52:34
1.3K0
发布2019-05-29 10:52:34
举报
文章被收录于专栏:北漂的我北漂的我

1. 配置 /opt/hadoop-2.7.4/etc/hadoop/core-site.xml

代码语言:javascript
复制
<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://bigdata:9000</value>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/home/bigdata/hadoopdata</value>
	</property>
	<property>
		<name>hadoop.proxyuser.bigdata.hosts</name>
		<value>*</value>
	</property>
	<property>
		<name>hadoop.proxyuser.bigdata.groups</name>
		<value>*</value>
	</property>
</configuration>

2. 配置 /opt/hadoop-2.7.4/etc/hadoop/yarn-site.xml

代码语言:javascript
复制
<configuration>
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.resourcemanager.address</name>
		<value>bigdata:18040</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.address</name>
		<value>bigdata:18030</value>
	</property>
	<property>
		<name>yarn.resourcemanager.resource-tracker.address</name>
		<value>bigdata:18025</value>
	</property>
	<property>
		<name>yarn.resourcemanager.admin.address</name>
		<value>bigdata:18141</value>
	</property>
	<property>
		<name>yarn.resourcemanager.webapp.address</name>
		<value>bigdata:18088</value>
	</property>
	<property>
		<name>yarn.resourcemanager.scheduler.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
	</property>
	<property>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
	</property>
	<property>
		<name>yarn.nodemanager.pmem-check-enabled</name>
		<value>false</value>
	</property>
	<property>
		<name>yarn.nodemanager.vmem-check-enabled</name>
		<value>false</value>
	</property>
</configuration>

3. 配置 /opt/hadoop-2.7.4/etc/hadoop/mapred-site.xml

代码语言:javascript
复制
<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.done-dir</name> 
		<value>/user/history/done</value> 
	</property> 
	<property> 
		<name>mapreduce.jobhistory.intermediate-done-dir</name> 
		<value>/user/history/done_intermediate</value> 
	</property>
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>bigdata:10020</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>bigdata:19888</value>
	</property>
</configuration>

4. 配置 /opt/hadoop-2.7.4/etc/hadoop/capacity-scheduler.xml (使用以下内容直接替换文件内容)

代码语言:javascript
复制
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare 
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare 
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>etl,report,default</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>40</value>
    <description>Default queue target capacity.</description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.etl.capacity</name>
    <value>30</value>
    <description>etl queue target capacity.</description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.report.capacity</name>
    <value>30</value>
    <description>report queue target capacity.</description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.etl.user-limit-factor</name>
    <value>1</value>
    <description>
      etl queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.report.user-limit-factor</name>
    <value>1</value>
    <description>
      report queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.etl.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the etl queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.report.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the report queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.etl.state</name>
    <value>RUNNING</value>
    <description>
      The state of the etl queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.report.state</name>
    <value>RUNNING</value>
    <description>
      The state of the report queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.etl.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the etl queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.report.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the report queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.etl.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the etl queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.report.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the report queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler 
      attempts to schedule rack-local containers. 
      Typically this should be set to number of nodes in the cluster, By default is setting 
      approximately number of nodes in one rack which is 40.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
    <description>
      A list of mappings that will be used to assign jobs to queues
      The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
      Typically this list will be used to map users to queues,
      for example, u:%user:%user maps all users to queues with the same name
      as the user.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
    </description>
  </property>

</configuration>

最后启动 yarn

1. 先启动 hdfs

代码语言:javascript
复制
/opt/hadoop-2.7.4/sbin/start-dfs.sh

2. 然后启动 yarn

代码语言:javascript
复制
/opt/hadoop-2.7.4/sbin/start-yarn.sh

访问 http://bigdata:18088 验证 yarn 是否启动成功

3. 最后启动 historyserver

代码语言:javascript
复制
/opt/hadoop-2.7.4/sbin/mr-jobhistory-daemon.sh start historyserver

访问 http://bigdata:19888 验证 historyserver 是否启动成功

本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档