Hadoop基础教程-第9章 HA高可用(9.4 YARN 高可用)(草稿)

第9章 HA高可用

9.4 YARN 高可用

9.4.1 RM单点故障

http://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html ResourceManager(RM)负责跟踪群集中的资源,并调度应用程序(例如MapReduce作业)。在Hadoop 2.4之前,ResourceManager是YARN集群中的单点故障。高可用性功能以活动/待机资源管理器对的形式添加冗余,以消除此单一故障点。

YARN高可用,也就是ResourceManager高可用,规划如下

IP

nodename

RM

NM

192.168.80.131

node1

Y

Y

192.168.80.132

node2

Y

192.168.80.133

node3

Y

Y

9.4.2 配置yarn-site.xml

[root@node1 hadoop]# vi yarn-site.xml
[root@node1 hadoop]# cat yarn-site.xml 
<?xml version="1.0"?>
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
        <description>NodeManager上运行的附属服务(运行MapReduce程序)</description>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
        <description>启用RM高可用性</description>
    </property>
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yarn1</value>
        <description>YARN集群ID</description>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
        <description>启用HA时,群集中的RM节点列表</description>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>node1</value>
        <description>第1个resourcemanager</description>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>node3</value>
        <description>第2个resourcemanager</description>
    </property>
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>node1:2181,node2:2181,node3:2181</value>
        <description>Zookeeper列表</description>
    </property>
</configuration>

[root@node1 hadoop]# 

9.4.3 修改yarn-env.sh

编译yarn-env.sh文件,增加PID文件存放目录

[root@node1 hadoop]# vi yarn-env.sh

添加一行,内容如下

export YARN_PID_DIR=/var/run

9.4.4 分发配置文件

[root@node1 hadoop]# scp yarn-site.xml node2:/opt/hadoop-2.7.3/etc/hadoop/
yarn-site.xml                                                                                                                                              100%  841     0.8KB/s   00:00    
[root@node1 hadoop]# scp yarn-site.xml node3:/opt/hadoop-2.7.3/etc/hadoop/
yarn-site.xml                                                                                                                                              100%  841     0.8KB/s   00:00     
[root@node1 hadoop]# scp yarn-env.sh node2:/opt/hadoop-2.7.3/etc/hadoop/
yarn-env.sh                                                                                                                                                100% 4595     4.5KB/s   00:00    
[root@node1 hadoop]# scp yarn-env.sh node3:/opt/hadoop-2.7.3/etc/hadoop/
yarn-env.sh                                                                                                                                                100% 4595     4.5KB/s   00:00    
[root@node1 hadoop]# 

9.4.5 启动YARN

[root@node1 hadoop]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-resourcemanager-node1.out
node2: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-node2.out
node3: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-node3.out
node1: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-node1.out
[root@node1 hadoop]# 

node1

[root@node1 hadoop]# jps
4337 Jps
4036 NodeManager
3189 DataNode
3529 NameNode
3402 DFSZKFailoverController
3931 ResourceManager
3310 JournalNode
[root@node1 hadoop]#

node2

[root@node2 ~]# jps
4306 Jps
3828 DFSZKFailoverController
3670 JournalNode
4200 NodeManager
3449 DataNode
3947 NameNode
[root@node2 ~]#

node3

[root@node3 ~]# jps
2643 JournalNode
2771 NodeManager
2549 DataNode
2871 Jps

单独启用备用resourcemanager

[root@node3 ~]# yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-resourcemanager-node3.out
[root@node3 ~]# jps
2643 JournalNode
2771 NodeManager
2900 ResourceManager
2932 Jps
2549 DataNode
[root@node3 ~]# 

9.4.6 Web

通过浏览器打开 http://192.168.80.131:8088

通过浏览器打开 http://192.168.80.133:8088 跳转到http://node1:8088/

实际上http://node1:8088/就是http://192.168.80.133:8088,是因为物理机Windows系统的hosts文件没有配置node1而已。

9.4.7 测试

[root@node1 hadoop]# jps
4337 Jps
4036 NodeManager
3189 DataNode
3529 NameNode
3402 DFSZKFailoverController
3931 ResourceManager
3310 JournalNode
[root@node1 hadoop]# kill 3931
[root@node1 hadoop]# 

这时再次打开http://192.168.80.133:8088

9.4.8 Zookeeper

[root@node2 ~]# zkCli.sh 
Connecting to localhost:2181
2017-07-22 10:36:18,876 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2017-07-22 10:36:18,881 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node2
2017-07-22 10:36:18,881 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_112
2017-07-22 10:36:18,884 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2017-07-22 10:36:18,885 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_112/jre
2017-07-22 10:36:18,885 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper-3.4.10/bin/../build/classes:/opt/zookeeper-3.4.10/bin/../build/lib/*.jar:/opt/zookeeper-3.4.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.10/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.10/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper-3.4.10/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.10/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.10/bin/../zookeeper-3.4.10.jar:/opt/zookeeper-3.4.10/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.10/bin/../conf:.::/opt/jdk1.8.0_112/lib
2017-07-22 10:36:18,885 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-07-22 10:36:18,885 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2017-07-22 10:36:18,885 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2017-07-22 10:36:18,885 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2017-07-22 10:36:18,885 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2017-07-22 10:36:18,886 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-514.el7.x86_64
2017-07-22 10:36:18,886 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2017-07-22 10:36:18,886 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2017-07-22 10:36:18,886 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
2017-07-22 10:36:18,888 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
Welcome to ZooKeeper!
2017-07-22 10:36:19,007 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2017-07-22 10:36:19,252 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-07-22 10:36:19,282 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x25d6a8d40030005, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper, test, yarn-leader-election, hadoop-ha]
[zk: localhost:2181(CONNECTED) 1] ls /yarn-leader-election
[yarn1]
[zk: localhost:2181(CONNECTED) 2] ls /yarn-leader-election/yarn1
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: localhost:2181(CONNECTED) 3] 

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏杨建荣的学习笔记

聊聊Data Guard中的DG Broker(r10笔记第24天)

DG Broker是Oracle为Data Guard维护提供的一个很不错的工具,从我的实际使用来看,早期的版本中似乎大家都还是存在一定的思维定式,认为手工维护...

2754
来自专栏Pythonista

有关gitlab的神秘操作.....version&&domain设置...

在使用gitlab的时候,如果服务器IP变动,之前的domain写入了配置文件了,如下路径:

1212
来自专栏Hadoop实操

如何使用Sentry管理Hive外部表权限

使用如下命令在HDFS的根目录下创建Hive外部表的数据目录/extwarehouse

9768
来自专栏数据库新发现

关于checkpoint cnt和checkpoint scn

SQL> alter session set events 'immediate trace name CONTROLF level 10';

1192
来自专栏个人分享

hadoop-spark-hive-hbase配置相关说明

或者 ./runRemoteCmd.sh '~/och200/zookeeper/bin/zkServer-initialize.sh --myid=1' zo...

1062
来自专栏杨建荣的学习笔记

一天内碰到的3个rac节点问题 (r6笔记第36天)

说到问题,真是层出不穷,自己也算搭建了也不少的rac环境的,但是在本地试验的时候总是会碰到一些问题,昨晚铲掉旧环境,搭建了两遍rac环境,终于在凌晨搭建好了环境...

3177
来自专栏Netkiller

ElasticSearch + Logstash + Kibana 日志采集

本文节选自《Netkiller Monitoring 手札》 ElasticSearch + Logstash + Kibana 一键安装 配置 logst...

2704
来自专栏乐沙弥的世界

非root用户tomcat daemon配置

基于安全策略来考虑,绝大多数应用程序都应以非root用户来启动,对于轻量级的应用程序,如tomcat,用root再寻常不过了。你懂的,方便啊。在生产环境这么干很...

692
来自专栏乐沙弥的世界

配置ORACLE 客户端连接到数据库

Oracle 客户端连接到Oracle 数据库服务器貌似不同于SQL serve中的网络配置,其实不然,只不过所有的SQL server 运行于Windows...

793
来自专栏Hadoop实操

如何在Redhat7.4的CDH5.15中启用Kerberos

2294

扫码关注云+社区