关于CDH5.13的安装的记录

原创

Bob hadoop

修改于 2021-02-19 11:47:51

6190

修改于 2021-02-19 11:47:51

文章被收录于专栏：日常杂记日常杂记

背景：写这篇文章是为了后面学习监控yarn的队列，Cloudera的CDH属于即将过时的技术了，但是目前国内很多企业任然以5.x 的版本为主，如果有不会的同学可以参考搭建一下。

软件版本： center os7.4 cdh5.13.1 cm5.13.1 mysql5.7 spark2.3

物理机配置

目前市面主流的物理机配置基本都为26核384g内存机器，盘的大小一般为4T*10

1、主机操作系统选择：center OS 7.4操作系统

2、主机swap配置

Linux系统中当内存使用到一定程度后会使用swap分区，这是由/proc/sys/vm/swappiness文件中的vm.swappiness 参数进行控制的，linux默认vm.swappiness=60

但是对于hadoop集群来说，如果使用系统默认设置，会导致swap分区被频繁使用，集群会不断发出警告。所以在搭建cloudera-manager的时候，会建议调整vm.swappiness参数，我一般设置为0，vm.swappiness的值越小使用swap分区的可能性就越小。设置方法为：

sysctl -w vm.swappiness=0

如果集群已经使用了swap分区，在设置vm.swappiness值之后建议对swap分区进行刷新,命令为：

swapoff -a 
swapon -a

也可在开机启动配置中直接增加swapoff -a

[hadoop@whx-9 ~]$ free -g
              total        used        free      shared  buff/cache   available
Mem:            125          24           8           0          92          99
Swap:             0           0           0

3、主机磁盘配置

对于所有已规划为datanode节点的机器的数据盘按照raid0 依次挂载在/mnt/data01-data10上，关于raid0与JBOD比较速度问题，笔者维护千节点集群暂无使用JBOD，但理论上JBOD吞吐速度好于raid方式。

主机测在做系统时有些盘是没有格式化与挂载的，可参考入戏操作进行格式化挂载。

for ip in `cat hostlist`; do ssh $ip  “sh  formatdisk.sh” ; done

格式化磁盘脚本formatdisk.sh

#!/bin/bash
for disk_id in {b..k}
do
        fdisk /dev/sd${disk_id} <<EOF
n
p
1
w
q
EOF
Done

生产机器默认10块磁盘，格式化完毕，挂载写入/etc/fstab中。

/dev/sdb1 /mnt/sd01 xfs defaults 0 0
/dev/sdc1 /mnt/sd02 xfs defaults 0 0
/dev/sdd1 /mnt/sd03 xfs defaults 0 0
/dev/sde1 /mnt/sd04 xfs defaults 0 0
/dev/sdf1 /mnt/sd05 xfs defaults 0 0
/dev/sdg1 /mnt/sd06 xfs defaults 0 0
/dev/sdh1 /mnt/sd07 xfs defaults 0 0
/dev/sdi1 /mnt/sd08 xfs defaults 0 0
/dev/sdj1 /mnt/sd09 xfs defaults 0 0
/dev/sdk1 /mnt/sd10 xfs defaults 0 0

for ip in `cat hostlist`; do scp  a.txt  $ip:/root/ ; done
for ip in `cat hostlist`; do ssh  $ip  “cat  a.txt >>/etc/fstab”; done

格式化磁盘vim disk.sh

#!bin/bash
for disk_id in {b..w}
do
        mkfs.xfs /dev/sd${disk_id}1
done

[hadoop@whx-8 ~]$ ll /mnt/ 
total 0
drwxr-xr-x 5 root root 83 May  7  2018 sd01
drwxr-xr-x 5 root root 83 May  7  2018 sd02
drwxr-xr-x 5 root root 83 May  7  2018 sd03
drwxr-xr-x 5 root root 83 May  7  2018 sd04
drwxr-xr-x 5 root root 83 May  7  2018 sd05
drwxr-xr-x 5 root root 83 May  7  2018 sd06
drwxr-xr-x 5 root root 83 May  7  2018 sd07
drwxr-xr-x 5 root root 83 May  7  2018 sd08
drwxr-xr-x 5 root root 83 May  7  2018 sd09
drwxr-xr-x 5 root root 83 May  7  2018 sd10
drwxr-xr-x 5 root root 83 May  7  2018 sd11

4、所有机器建议增加hadoop用户，为后续运维使用

此处需要有大量规范化项目，如果公司集群很大时建议规范建设，譬如技术部（Technical department）下边分有开发组，运维组，各组又按照项目分裂，需要考虑好。

[hadoop@whx-8 ~]$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
avahi-autoipd:x:170:170:Avahi IPv4LL Stack:/var/lib/avahi-autoipd:/sbin/nologin
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
systemd-network:x:998:996:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:997:995:User for polkitd:/:/sbin/nologin
tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin
unbound:x:996:994:Unbound DNS resolver:/etc/unbound:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
chrony:x:995:993::/var/lib/chrony:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
imonitor:x:1003:1004::/home/imonitor:/bin/bash
deployer:x:1004:1005::/home/deployer:/bin/bash
maintainer:x:1005:1006::/home/maintainer:/bin/bash
ntp:x:38:38::/etc/ntp:/sbin/nologin
weihu:x:1007:1008::/home/weihu:/bin/bash
hadoop:x:1008:1009::/home/hadoop:/bin/bash
telegraf:x:1012:1012::/home/telegraf:/bin/bash

5、主机名配置

建议统一配置主机名，其中主机名不建议添加大写和特殊字符，主要和Hadoop安全认证有关。

6、机架配置

机架的配置对Hadoop集群也是有一定的影响的，hadoop在分配MR到各个节点时会倾向于执行机架内的数据传输，而非跨机架数据传输，hdfs还能够更加智能的放置副本，以取得性能和弹性的平衡。

软件配置

1、修改hosts

一批新机器采购上架以后需要安装操作系统，如果用于CDH集群的建设时要注意：新机器主机名称不要出现大写字母如DSJ字母，会在cm页面出现配置告警，与安全认证有关。如果集群启用kerberos时需要注意此事项

[root@whx-1 ~]# cat /etc/hosts
127.0.0.1    localhost
192.168.1.1  whx-1
192.168.1.2  whx-2
192.168.1.3  whx-3
192.168.1.4  whx-4
192.168.1.5  whx-5
192.168.1.6  whx-6
192.168.1.7  whx-7

2、配置免密码登陆

由于集群规模超大所有机器建议免密登陆，待集群配置安装完毕后建议回收取消root免密，免密配置如下：

[root@whx-1 桌面]# ssh-keygen -t rsa -P ""
[root@whx-1 ~]# cd .ssh
[root@whx-1 .ssh]# ls
id_rsa  id_rsa.pub
[root@whx-1 .ssh]# ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.1.1

3、关闭防火墙selinux

集群运行时需要关闭所有节点防火墙以及selinux，生产集群不要暴漏在外网中。

[root@whx-1 ~]# systemctl stop firewalld.service && systemctl disable firewalld.service
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

可批量关闭，在root家目录创建hostlist文件里面为所有机器ip。

for ip in `cat hostlist`; do ssh $ip "systemctl stop firewalld.service && systemctl disable firewalld.service"; done

关闭selinux

及时生效

[root@whx-1 ~]# setenforce 0
[root@whx-1 ~]# getenforce
Permissive

永久生效

[root@whx-1 ~]# vim /etc/selinux/config
SELINUX=enforcing改成disabled

批量修改

for ip in `cat hostlist` ; do scp /etc/selinux/config root@$ip:/etc/; done
for ip in `cat hostlist`; do ssh $ip "setenforce 0"; done

4、安装jdk

查看机器是否安装jdk

这里建议安装oracle的jdk1.8

首先卸载自带open jdk

[root@whx-1 ~]# rpm -aq |grep jdk
java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64
java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64

批量卸载

for ip in  `cat iplist`; do ssh $ip "rpm -e java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64 --nodeps && rpm -e java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64 --nodeps"; done

下载oracle的jdk解压到/usr/java/下

在/etc/profile追加如下，source一下

PATH=$PATH:/usr/java/jdk1.8.0_161
export JAVA_HOME=/usr/java/jdk1.8.0_161
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

此处说明下，当集群需要多个版本java时建议不要修改全局变量，可配置用户家目录的.bashrc文件，自定义java路径在/opt/soft/java/下。当用户使用

5、安装mysql数据库

建议使用mysql-mha模式，以下为单节点mysql配置方法。

[root@whx-1 opt]# tar xvf mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar
mysql-community-server-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-client-5.7.18-1.el7.x86_64.rpm
mysql-community-common-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-5.7.18-1.el7.x86_64.rpm
mysql-community-server-minimal-5.7.18-1.el7.x86_64.rpm（这个包可以删除）
mysql-community-test-5.7.18-1.el7.x86_64.rpm
mysql-community-minimal-debuginfo-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm
[root@whx-1 opt]# rm mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar -f
[root@whx-1 opt]# mkdir mysql
[root@whx-1 opt]# mv mysql-community-* mysql
[root@whx-1 opt]# cd mysql/
[root@whx-1 mysql]# createrepo ./
Spawning worker 0 with 12 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

配置mysql的yum源，该步骤主要是方便其他机器安装mysql。

[root@whx-1 mysql]# cd /etc/yum.repos.d/
[root@whx-1 yum.repos.d]# vim mysql.repo
[mysql]
name=5.7
baseurl=file:///opt/mysql/
enable=1
gpgcheck=0
[root@whx-1 yum.repos.d]# yum repolist
源标识                               源名称                                状态
mysql                                5.7                                    12
repolist: 12

卸载mariadb的lib

[root@whx-1 yum.repos.d]# rpm -qa | grep mariadb
mariadb-libs-5.5.44-2.el7.centos.x86_64
[root@whx-1 yum.repos.d]# rpm -e mariadb-libs-5.5.44-2.el7.centos.x86_64 –nodeps

安装mysql数据库，此处可注释掉本地center os 7的yum源

[root@whx-1 yum.repos.d]# yum -y install mysql
[root@whx-1 yum.repos.d]# yum -y install mysql-server

启动服务

[root@whx-1 log]# systemctl start mysqld.service
[root@whx-1 log]# cat  /var/log/mysqld.log 
2021-02-18T12:03:18.062358Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-02-18T12:03:18.198096Z 0 [Warning] InnoDB: New log files created, LSN=45790
2021-02-18T12:03:18.236126Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2021-02-18T12:03:18.293010Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 49b23bd5-71e1-11eb-b737-000c29cadc9d.
2021-02-18T12:03:18.294346Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2021-02-18T12:03:18.295554Z 1 [Note] A temporary password is generated for root@localhost: Rieh!a-d,9JH
2021-02-18T12:03:20.706153Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-02-18T12:03:20.708998Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.18) starting as process 54291 ...
2021-02-18T12:03:20.713135Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-02-18T12:03:20.713181Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-02-18T12:03:20.713190Z 0 [Note] InnoDB: Uses event mutexes
2021-02-18T12:03:20.713198Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-02-18T12:03:20.713211Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-02-18T12:03:20.713219Z 0 [Note] InnoDB: Using Linux native AIO
2021-02-18T12:03:20.713635Z 0 [Note] InnoDB: Number of pools: 1
2021-02-18T12:03:20.713822Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-02-18T12:03:20.716317Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-02-18T12:03:20.728818Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-02-18T12:03:20.732664Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-02-18T12:03:20.749189Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-02-18T12:03:20.767208Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-02-18T12:03:20.767281Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-02-18T12:03:20.784538Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-02-18T12:03:20.788882Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2021-02-18T12:03:20.788900Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2021-02-18T12:03:20.791200Z 0 [Note] InnoDB: Waiting for purge to start
2021-02-18T12:03:20.841872Z 0 [Note] InnoDB: 5.7.18 started; log sequence number 2535704
2021-02-18T12:03:20.842631Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-02-18T12:03:20.843650Z 0 [Note] Plugin 'FEDERATED' is disabled.
2021-02-18T12:03:20.848107Z 0 [Note] InnoDB: Buffer pool(s) load completed at 210218 20:03:20
2021-02-18T12:03:20.856534Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2021-02-18T12:03:20.856776Z 0 [Warning] CA certificate ca.pem is self signed.
2021-02-18T12:03:20.857903Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2021-02-18T12:03:20.857949Z 0 [Note] IPv6 is available.
2021-02-18T12:03:20.857977Z 0 [Note]   - '::' resolves to '::';
2021-02-18T12:03:20.857995Z 0 [Note] Server socket created on IP: '::'.
2021-02-18T12:03:20.879682Z 0 [Note] Event Scheduler: Loaded 0 events
2021-02-18T12:03:20.879969Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.18'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2021-02-18T12:03:20.879991Z 0 [Note] Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine. You may use the startup option '--disable-partition-engine-check' to skip this check. 
2021-02-18T12:03:20.880009Z 0 [Note] Beginning of list of non-natively partitioned tables
2021-02-18T12:03:20.900544Z 0 [Note] End of list of non-natively partitioned tables

查看mysql的root用户初始密码

[root@whx-1 log]# grep 'temporary password' /var/log/mysqld.log
2021-02-18T12:03:18.295554Z 1 [Note] A temporary password is generated for root@localhost: Rieh!a-d,9JH
[root@bigdata01 log]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3

Server version: 5.7.18
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

进入后无法进行其他操作需要先修改mysql服务的root密码。
mysql>  flush PRIVILEGES;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

由于5.7以后的mysql密码复杂度有设置需求所以无法设置弱密码。
mysql> alter user 'root'@'localhost' identified by '123456';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

设置复杂度较高的密码
mysql> alter user 'root'@'localhost' identified by '1qaz@WSX';
Query OK, 0 rows affected (0.00 sec)

授权远程root登陆的密码
mysql> grant all privileges  on *.* to 'root'@'%' identified by '1qaz#EDC' with grant option;
Query OK, 0 rows affected, 1 warning (0.00 sec)

查看root账号的授权的状态
mysql> show grants for root@'%' ;
+-------------------------------------------------------------+
| Grants for root@%                                           |
+-------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION |
+-------------------------------------------------------------+
1 row in set (0.00 sec)

建立数据库用于后边服务安装，如果不想复用数据库可不建立。

mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

6、配置ntp服务

正常生产集群环境中会有时钟源，将时钟源指定一下即可。此处大致掠过

[root@whx-1 yum.repos.d]# yum -y install ntp
[root@whx-1 yum.repos.d]# chkconfig ntpd on
[root@whx-1 yum.repos.d]# service ntpd start

7、所有节点创建用户

[root@whx-1 yum.repos.d]# useradd --system --home=/opt/cm-5.13.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

集群安装

1、下载CM与CDH

CM地址：http://archive.cloudera.com/cm5/cm/5/
CDH地址：http://archive.cloudera.com/cdh5/parcels/5.13.1/

所需文件为

CDH-5.13.1-1.cdh5.13.1.p0.2-el7.parcel     
cloudera-manager-centos7-cm5.13.1_x86_64.tar.gz
CDH-5.13.1-1.cdh5.13.1.p0.2-el7.parcel.sha 
manifest.json

注意sha1要改成sha

2、分发软件包

解压cm

[root@whx-1 opt]# tar xvf cloudera-manager-centos7-cm5.13.1_x86_64.tar.gz

会生成cm-5.13.1与cloudera两个文件夹

修改mysql的jdbc驱动移动到对应的目录

[root@whx-1 opt]# mv mysql-connector-java-5.1.46.jar mysql-connector-java.jar
[root@whx-1 opt]# cp mysql-connector-java.jar /usr/share/java/
[root@whx-1 opt]# cp mysql-connector-java.jar /opt/cm-5.13.1/share/cmf/lib/

初始化数据库

 /opt/cm-5.13.1/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm
Enter database password: 此处是123456
Enter SCM password:此处是passwd

将parcel包移动到对应目录

[root@whx-1 opt]# mv CDH-5.13.1-1.cdh5.13.1.p0.2-el7.parcel* cloudera/parcel-repo/
[root@whx-1 opt]# mv manifest.json cloudera/parcel-repo/

修改agent文件

[root@whx-1 opt]# vim /opt/cm-5.13.1/etc/cloudera-scm-agent/config.ini
server_host=whx-1

分发其他节点

[root@whx-1 opt]# for i in `cat /root/iplist`;do scp /opt/cm-5.13.1/etc/cloudera-scm-agent/config.ini $i:/opt/cm-5.13.1/etc/cloudera-scm-agent/;done

3、启动server和agent

[root@whx-1 opt]# /opt/cm-5.13.1/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  确定  ]
[root@whx-1 opt]# /opt/cm-5.13.1/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  确定  ]