CDH 6.3.2 安装（一）

静谧星空TEL

发布于 2021-04-27 11:37:40

2.8K0

发布于 2021-04-27 11:37:40

文章被收录于专栏：云计算与大数据技术

官方文档：https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/installation_reqts.html

notepad++：https://notepad-plus-plus.org/downloads/v7.8.5/

xshell：https://www.netsarang.com/zh/xshell/

everthing：https://www.voidtools.com/Everything-1.4.1.969.x64-Setup.exe

jdk1.8： oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

mysql5.7.28：mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar

mysql5.7.29：mysql-5.7.29-linux-glibc2.12-x86_64.tar

1、安装通用依赖

yum update

yum -y install vim wget lrzsz net-tools psmisc iptables-services

2、主机名配置

hostnamectl set-hostname cdh1

vim /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=cdh1

cat /etc/sysconfig/network

3、网络配置

vim /etc/hosts

cat /etc/hosts

cp /etc/hosts /etc/hosts.template

在本地编辑一个hosts文件，上传至 /etc/ 目录下

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.232.131 cdh1
192.168.232.132 cdh2
192.168.232.133 cdh3
192.168.232.134 cdh4
192.168.232.135 cdh5
192.168.232.136 cdh6

在Windows下的host文件追加

# cdh company cluster
# 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
# ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.232.131 cdh1
192.168.232.132 cdh2
192.168.232.133 cdh3
192.168.232.134 cdh4
192.168.232.135 cdh5
192.168.232.136 cdh6

4、禁用ipv6网络协议

CentOS7 mini版本没有安装 ipconfig模块，需要用户手动安装

yum -y install net-tools

安装pstree模块

yum -y install psmisc

树形图显示父子进程：

pstree -aup

禁用IPV6地址

cp /etc/sysctl.conf /etc/sysctl.conf.template

echo -e "
# 禁用整个系统所有接口的IPv6
net.ipv6.conf.all.disable_ipv6 = 1
# 禁用某一个指定接口的IPv6（eth0，lo）
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
" >> /etc/sysctl.conf

重启sysctl：

sudo sysctl -p

再次查看网络协议

ifconfig

查看防火墙状态

systemctl status firewalld.service

安装iptables-services模块

yum -y install iptables-services

关闭防火墙

service iptables stop

查看状态

service iptables status

关闭防火墙，禁止开机自启

systemctl stop firewalld.service

systemctl disable firewalld.service

关闭SELINUX

setenforce 0

vim /etc/selinux/config

把 SELINUX=enforcing 改成 SELINUX=disable

重启服务器

reboot

下载 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

mkdir /usr/java

cd /usr/java/

将下载的 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 上传到 /usr/java目录下，并解压

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

删除文件 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

rm -rf oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

创建软链接

ln -s /usr/java/jdk1.8.0_181-cloudera /usr/java/jdk1.8

查看软链接

配置环境变量 /etc/profile

cp /etc/profile /etc/profile.template

echo '
export JAVA_HOME=/usr/java/jdk1.8
export MYSQL_HOME=/usr/local/mysql
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=${JAVA_HOME}/bin:${PATH}:${CLASSPATH}:${MYSQL_HOME}/bin
' >> /etc/profile

tail -n 10 /etc/profile

source /etc/profile

java -version

which java

安装 NTP 模块

yum -y install ntp

设置开机自启

chkconfig ntpd on

查看ntp是否开机自启

systemctl is-enabled ntpd

设置 NTP 服务

ntpdate -u ntp.aliyun.com

半个小时同步一次（非必须）

crontab -e

0,30 * * * * ntpdate -u ntp.aliyun.com

或者将命令添加到 /etc/crontab文件中（非必须）

echo '0,30 * * * * root ntpdate -u ntp.aliyun.com' >> /etc/crontab
cat /etc/crontab

查看时间是否准确

date

安装MySQL数据库

CDH官方安装文档：https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#cmig_topic_5_5

cd /usr/local

使用SFTP工具（大文件传输速度更快更稳定），上传MySQL包

cd /usr/local/

先解压 tar 文件

tar -xvf mysql-5.7.29-linux-glibc2.12-x86_64.tar

再解压 tar.gz 文件

tar -zxvf mysql-5.7.29-linux-glibc2.12-x86_64.tar.gz

删除 tar文件和 gz文件

rm -rf /usr/local/*.tar /usr/local/*.gz

重命名 mysql-5.7.29-linux-glibc2.12-x86_64 为 mysql

mv mysql-5.7.29-linux-glibc2.12-x86_64/ mysql

检查是否有老版本

如果没有任何提示信息则表示Linux上没有安装mysql，如果有会显示mysql的版本

rpm -qa | grep mysql

创建mysql用户组和用户并修改权限

groupadd mysql
useradd -r -g mysql mysql

创建数据目录并赋权

mkdir -p /data/mysql
chown mysql:mysql -R /data/mysql

查看mysql配置信息

cat /etc/my.cnf

配置文件前，先关闭MySQL服务

sudo systemctl stop mysqld

配置 /etc/my.cnf 文件

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 1000
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

初始化数据库，进入mysql的bin目录

mv mysql-5.7.29-linux-glibc2.12-x86_64 mysql

cd /usr/local/mysql/bin/

初始化 MySQL

./mysqld --defaults-file=/etc/my.cnf --basedir=/usr/local/mysql --datadir=/data/mysql/
--socket=/data/mysql/mysql.sock log-error=/data/mysql/mysqld.log --pid-file=/data/mysql/mysql.pid --user=root  --initialize

查看临时密码

cat /data/mysql/mysql.err

启动MySQL

service mysql start

ps -ef|grep mysql

修改MySQL密码

cd /usr/local/mysql/bin/

./mysql -uroot -p

set password=PASSWORD('root');

将 /etc/profile 文件下载到本地，方便修改

sz /etc/profile

配置 MySQL环境变量

vim /etc/profile

将之前配置的Java环境变量注释掉，重新配置Java和MySQL环境变量

# export JAVA_HOME=/usr/java/jdk1.8
# export CLASSPATH=.:$JAVA_HOMdE/lib/dt.jar:$JAVA_HOME/lib/tools.jar
# export PATH=$JAVA_HOME/bin:$PATH
JAVA_HOME=/usr/java/jdk1.8
CLASSPATH=.:$JAVA_HOMdE/lib/dt.jar:$JAVA_HOME/lib/tools.jar
MYSQL_HOME=/usr/local/mysql

PATH=$PATH:$JAVA_HOME/bin:$CLASSPATH:$MYSQL_HOME/bin

export JAVA_HOME PATH CLASSPATH MYSQL_HOME

设置PATH的语法都为：

export PATH=$PATH::::------:

重新加载配置文件

source /etc/profile

重新验证环境变量

java -verion
mysql -uroot -proot

设置mysql开机自启

首先查看开机自启的程序

chkconfig --list

将mysql服务添加到开机自启选项

把mysql的启动脚本放到开机初始化目录

cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld

对文件赋予执行权限

chmod +x /etc/init.d/mysqld 或 chmod 777 /etc/init.d/mysqld

增加mysqld服务

chkconfig --add mysqld

查询mysqld服务情况

chkconfig --list mysqld

开启运行级别语法（不需要执行）

chkconfig --level 345 mysqld on

重启服务器验证：reboot

给 root 用户赋权

grant all privileges on *.* to root@'%' identified by 'root';

添加远程访问权限

uer mysql;
update user set host='%' where user = 'root';

发现报错：ERROR 1062 (23000): Duplicate entry '%-root' for key 'PRIMARY'

查看 mysql.user 表的信息，发现 root 用户已经有了远程访问的权限

select user,host from user;

刷新权限，重启MySQL

flush privileges;

service mysql restart

创建大数据集群所需要的数据库

##DROP DATABASE scm;
##DROP USER scm;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;


GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'root';

flush privileges;

安装 CDH 前先把需要的依赖包下载好

yum -y install chkconfig httpd vim wget lrzsz net-tools psmisc iptables-services gcc gcc-c++ fop lsb bison flex readline-devel zlib-devel  python libxslt-devel python-devel psmisc zlib sqlite cyrus-sasl-plain postgresql-libs.x86_64 cyrus-sasl-gssapi fuse fuse-libs redhat-lsb openssl-devel docbook-dtds docbook-style-xsl  libxslt perl bind-utils portmap mod_ssl python-psycopg2 MySQL-python

安装HTTP服务

CM的管理方式是通过Web请求访问管理的，需要在主节点上安装HTTP服务，并且启动开机自启服务

yum -y install httpd
chkconfig httpd on
service httpd start

查看是否开机自启

systemctl is-enabled httpd.service
systemctl is-active httpd

如果为disable，则设置开机自启

systemctl enable httpd.service

这里还有一个官网推荐的Linux配置，直接粘上去就行了

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /etc/rc.local 

sysctl vm.swappiness=10
echo 'vm.swappiness=10'>> /etc/sysctl.conf
cat /proc/sys/vm/swappiness

yum -y install krb5-workstation krb5-libs

至此，CDH 安装前的环境变量已经配好了

接下来直接复制虚拟机安装Cloudera Manager Server&Agent

CDH 6.3.2 安装（一）：https://blog.csdn.net/qq262593421/article/details/105048233

CDH 6.3.2 安装（二）：https://blog.csdn.net/qq262593421/article/details/105173259

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2020/03/24 ，如有侵权请联系 cloudcommunity@tencent.com 删除

云数据库 SQL Server