0728-6.3.0-如何在Redhat7.4安装CDH6.3

文档编写目的

Cloudera在2019年7月31日,对外宣布正式发布Cloudera Enterprise 6.3,相关介绍可以参考Fayson之前的文章《0682-Cloudera Enterprise 6.3.0发布》,《0717-6.3.0-Cloudera Manager 6.3的新功能》和《0718-6.3.0-CDH6.3的新功能》。本文档Fayson主要描述如何在Redhat7.4安装CDH6.3。CDH6与CDH5的安装步骤一致,主要包括以下四部分:

1.安全前置准备,包括安装操作系统、关闭防火墙、同步服务器时钟等;

2.外部数据库如MySQL安装

3.安装Cloudera Manager;

4.安装CDH集群;

请务必注意CDH6的安装前置条件包括如下:

  • 外部数据库支持:

MySQL 5.7或更高

MariaDB 5.5或更高

PostgreSQL 8.4或更高

Oracle 12c或更高

  • JDK

Oracle JDK1.8,将不再支持JDK1.7

  • 操作系统支持

RHEL 6.8或更高

RHEL 7.2或更高

SLES 12 SP2或更高

Ubuntu 16或更高

  • 本次Fayson的测试环境为

1.CM和CDH版本为6.3

2.Redhat7.4

3.JDK1.8.0_181

4.MariaDB-5.5.56

5.root用户安装

前置准备

2.1 hostname及hosts配置

集群中各个节点之间能互相通信使用静态IP地址。IP地址和主机名通过/etc/hosts配置,主机名通过/etc/hostname进行配置。

以cm节点(172.31.13.38)为例:

  • hostname配置

/etc/hostname文件如下:

ip-172-31-13-38.ap-southeast-1.compute.internal

或者你可以通过命令修改立即生效

ip-172-31-13-38.ap-southeast-1.compute.internal
  • hosts配置

/etc/hosts文件如下:

[root@ip-172-31-13-38 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.31.13.38 ip-172-31-13-38.ap-southeast-1.compute.internal
172.31.11.232 ip-172-31-11-232.ap-southeast-1.compute.internal
172.31.11.9 ip-172-31-11-9.ap-southeast-1.compute.internal
172.31.13.166 ip-172-31-13-166.ap-southeast-1.compute.internal

以上两步操作,在集群中其它节点做相应配置。确认需要安装的4台主机的hosts文件:

2.2 禁用SELinux

在所有节点执行setenforce 0 命令,此处使用批处理shell执行:

[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "setenforce 0"

集群所有节点修改/etc/selinux/config文件如下:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

2.3 关闭防火墙

集群所有节点执行 systemctl stop命令,此处通过shell批量执行命令如下:

[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl stop firewalld"
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl disable firewalld"
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl status firewalld"

2.4 集群时钟同步

在Redhat7.x的操作系统上,已经默认的安装了chrony,我们这里先卸载chrony,然后安装ntp。使用ntp来配置各台机器的时钟同步,将cm(172.31.13.38)服务作为本地ntp服务器,其它3台服务器与其保持同步。

1.所有机器卸载chrony

[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl stop firewalld"
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl disable firewalld"
[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl status firewalld"

2.所有机器安装ntp

[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "yum -y install ntp"

3.cm机器配置时钟与自己同步

[root@ ip-172-31-13-38 shell]# vim /etc/ntp.conf
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

4.集群其它节点,配置找cm机器去同步

[root@ ip-172-31-13-38 shell]# vim /etc/ntp.conf
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

5.重启所有机器的ntp服务

[root@ ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl restart ntpd"
[root@ ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl enable ntpd"
[root@ ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "systemctl status ntpd"

6.验证始终同步,在所有节点执行ntpq -p命令,如下使用脚本批量执行

[root@ip-172-31-13-38 shell]# sh ssh_do_all.sh node.list "ntpq -p"

左边出现*号表示同步成功。

2.5 设置swap

1.在所有机器执行以下命令以临时设置swap为1,并即时生效

sh ssh_do_all.sh node.list "sysctl -a | grep vm.swappiness"
sh ssh_do_all.sh node.list "echo 1 > /proc/sys/vm/swappiness"
sh ssh_do_all.sh node.list "sysctl -a | grep vm.swappiness"

确保最后swap打印应该都为1

2.为所有机器永久设置swap为1,修改/etc/sysctl.conf中vm.swappiness为1,没有则新增。

[root@ip-172-31-13-38 ~]# vim /etc/sysctl.conf
vm.swappiness = 1

将/etc/sysctl.conf文件同步到集群所有机器

2.6 设置透明大页面

1.所有节点执行以下命令关闭透明大页面,并即时生效

sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
sh ssh_do_all.sh node.list  "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
sh ssh_do_all.sh node.list  "cat /sys/kernel/mm/transparent_hugepage/enabled"
sh ssh_do_all.sh node.list "cat /sys/kernel/mm/transparent_hugepage/defrag"

2.修改所有节点的/etc/rc.d/rc.local文件的权限以实现开机执行

sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
sh ssh_do_all.sh node.list  "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
sh ssh_do_all.sh node.list  "cat /sys/kernel/mm/transparent_hugepage/enabled"
sh ssh_do_all.sh node.list "cat /sys/kernel/mm/transparent_hugepage/defrag"

3.在所有节点的/etc/rc.d/rc.local文件中新增如下内容,以实现开机自动关闭透明大页面。

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi

将该文件同步到所有节点

2.7 配置操作系统repo

Fayson用的是AWS的环境,这步是可以省略的,放在这里供物理机部署的兄弟们参考。

  • 挂载操作系统iso文件
[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /media/DVD1
[ec2-user@ip-172-31-2-159 ~]$ sudo mount -o loop 
CentOS-7-x86_64-DVD-1611.iso /media/DVD1/
  • 配置操作系统repo
[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/local_os.repo
[local_iso]    
name=CentOS-$releasever - Media
baseurl=file:///media/DVD1
gpgcheck=0
enabled=1
[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

2.8 安装httpd服务

  • 安装httpd服务
[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install httpd
  • 启动httpd服务
[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start httpd
  • 安装完httpd后,重新制作操作系统repo,换成http的方式方便其它服务器也可以访问
[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /var/www/html/iso
[ec2-user@ip-172-31-2-159 ~]$ sudo scp -r /media/DVD1/* /var/www/html/iso/
[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/os.repo 
[osrepo]
name=os_repo
baseurl=http://172.31.2.159/iso/
enabled=true
gpgcheck=false
[ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist

1.修改/etc/httpd/conf/httpd.conf配置文件,在<IfModule mime_module>中修改以下内容

AddType application/x-gzip .gz .tgz .parcel

2.保存httpd.conf的修改,并重启httpd服务

[root@ip-172-31-13-38 java]# systemctl restart httpd

2.9 安装MariaDB

1.安装MariaDB

[root@ip-172-31-13-38 ~]# yum -y install mariadb
[root@ip-172-31-13-38 ~]# yum -y install mariadb-server

2.启动并配置MariaDB

[root@ip-172-31-13-38 ~]# systemctl start mariadb
[root@ip-172-31-13-38 ~]# systemctl enable mariadb
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@ip-172-31-13-38 ~]# /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

3.建立CM,Hive等需要的表

[root@ip-172-31-13-38 ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 5.5.64-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database metastore default character set utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database cm default character set utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'cm'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database am default character set utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'am'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON am. * TO 'am'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database rm default character set utf8;
Query OK, 1 row affected (0.01 sec)

MariaDB [(none)]> CREATE USER 'rm'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database hue default character set utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'hue'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database oozie default character set utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database sentry default character set utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'sentry'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database nav_ms default character set utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'nav_ms'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON nav_ms. * TO 'nav_ms'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database nav_as default character set utf8;
Query OK, 1 row affected (0.01 sec)

MariaDB [(none)]> CREATE USER 'nav_as'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON nav_as. * TO 'nav_as'@'%';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]>

4.安装JDBC驱动

[root@ip-172-31-13-38 shell]# mkdir -p /usr/share/java/
[root@ip-172-31-13-38 ~]# mv mysql-connector-java-5.1.34.jar /usr/share/java/
[root@ip-172-31-13-38 ~]# cd /usr/share/java
[root@ip-172-31-13-38 java]# chmod 777 mysql-connector-java-5.1.34.jar
[root@ip-172-31-13-38 java]# ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar
[root@ip-172-31-13-38 java]# ll
total 940
-rwxrwxrwx. 1 root root 960372 May 16 15:53 mysql-connector-java-5.1.34.jar
lrwxrwxrwx. 1 root root     31 May 16 15:53 mysql-connector-java.jar -> mysql-connector-java-5.1.34.jar

Cloudera Manager安装

3.1 配置本地repo源

1.下载CM6.3的安装包,地址为:

https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.3.0-1281944.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-6.3.0-1281944.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
https://archive.cloudera.com/cm6/6.3.0/allkeys.asc

2.下载CDH6.3的安装包,地址为:

https://archive.cloudera.com/cdh6/6.3.0/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel
https://archive.cloudera.com/cdh6/6.3.0/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha1
https://archive.cloudera.com/cdh6/6.3.0/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha256
https://archive.cloudera.com/cdh6/6.3.0/parcels/manifest.json

3.将Cloudera Manager安装需要的6个rpm包以及一个asc文件下载到本地,放在同一目录,执行createrepo命令生成rpm元数据。

[root@ip-172-31-13-38 cm6.3]# createrepo .

4.配置Web服务器

将上述cdh6.3/cm6.3目录移动到/var/www/html目录下, 使得用户可以通过HTTP访问这些rpm包。

[root@ip-172-31-13-38 ~]# mv cm6.3/ cdh6.3/ /var/www/html/
[root@ip-172-31-13-38 ~]# cd /var/www/html/
[root@ip-172-31-13-38 html]# ll  *6.3

验证浏览器能否正常访问

5.制作Cloudera Manager的repo源

[root@ip-172-31-13-38 yum.repos.d]# vim cm.repo
[cmrepo]
name = cm_repo
baseurl = http://172.31.13.38/cm6.3
enable = true
gpgcheck = false
[root@ip-172-31-13-38 yum.repos.d]# yum repolist

6.验证安装JDK

[root@ip-172-31-13-38 yum.repos.d]# yum -y install oracle-j2sdk1.8-1.8.0+update181-1.x86_64

3.2 安装Cloudera Manager Server

1.通过yum安装Cloudera Manager Server

[root@ip-172-31-13-38 java]# yum -y install cloudera-manager-server

2.初始化数据库

[root@ip-172-31-13-38 java]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm password

3.启动Cloudera Manager Server

[root@ip-172-31-13-38 ~]# systemctl start cloudera-scm-server
[root@ip-172-31-13-38 ~]# systemctl status cloudera-scm-server

4.检查端口是否监听

[root@ip-172-31-13-38 ~]# netstat -lnpt | grep 7180
tcp        0      0 0.0.0.0:7180            0.0.0.0:*               LISTEN      22048/java

5.通过http://cm_ip:7180/cmf/login访问CM

注意:完全启动成功需要几分钟时间。

CDH安装

4.1 CDH集群安装向导

1.admin/admin登录到CM

2.同意license协议,点击继续

3.选择60试用,点击继续

4.点击“继续”

5.点击“继续”,输入集群名称,可以使用默认的“Cluster 1”。

6.输入主机IP或者名称,点击搜索找到主机后点击继续

7.选择自定义存储库,输入cm的http地址

CDH and other software”中选择“使用 Parcel (建议)”,点击“ 更多选项”,点击“-”删除其它所有地址,输入http://172.31.13.38/cdh6.3,点击“保存更改”

点击“继续”

8.点击“继续”,进入下一步安装jdk

9.点击“继续”,进入下一步配置ssh账号密码

10.点击“继续”,进入下一步,安装Cloudera Manager相关到各个节点

等待Agent安装完毕后,自动跳转到下一步开始分发Parcel

11.点击“继续”,进入下一步安装cdh到各个节点

12.点击Inspect Network Performance和Inspect Hosts,检查主机

如果有错误或者黄色警告,查看“显示检查器结果”,并逐项解决,然后“重新运行”检查,直到所有的检查都通过,否则没办法点击继续下一步。

4.2 集群设置安装向导

1.选择需要安装的服务,根据需要选择,这里随便选择Data Warehouse,也可以自定义服务

2.点击“继续”,进入集群角色分配,一台机器作为管理节点,另外三台机器作为DataNode

注意:Activity Monitor和Telemetry Publisher不用选择任何主机,留空,即不安装,因为用不到。

3.点击“继续”,进入下一步,测试数据库连接

4.测试成功,点击“继续”,进入目录设置,此处使用默认默认目录,根据实际情况进行目录修改

5.点击“继续”,进入各个服务启动

6.安装成功后进入home管理界面

4.3 组件版本检查

可以看到Hadoop3.0,Flume1.9,HBase2.1,Hive2.1.1,Spark2.4,Hue4.2.0,Impala3.2,Kafka2.2.1,Kudu1.0,Oozie5.1,Pig0.17,Senty2.1,Solr7.4,Sqoop1.4.7,Zookeeper3.4.5等。

总结

1.从安装方式上来看,CDH6与CDH5变化不大,这也方便了CDH5的用户可以较为快速的迁移到CDH6,以及适应CDH6的安装与使用。

2.安装向导界面有一些变化,现在可以一目了然的看到一共多少步骤,以及每个步骤是干什么。

3.安装条件前置没有任何变化,包括防火墙,Selinux关闭,ntp同步等等。可以参考Fayson之前的文章《CDH安装前置准备

4.进到主界面变化也不大,主要是Cloudera的logo变成了黑色,与Cloudera主页的整体风格一致。

5.在配置Cloudera Manager连接到数据库时的脚本有所变化。以前是/usr/share/cmf/schema/scm_prepare_database.sh,现在是/opt/cloudera/cm/schema/scm_prepare_database.sh

6.Cloudera Manager服务的状态在Redhat7通过systemctl status cloudera-scm-server查看是显示正确,而以前是不正确的,可以参考Fayson之前的文章《Cloudera Manager服务在RedHat7状态显示异常分析

7.Cloudera Manager的rpm安装包由之前的7个变成了6个,去掉了之前的JDK6的包,然后自带JDK1.8.0_181,将不再支持JDK1.7。

8.注意CM的安装除了下载rpm包以外,还要下载allkeys.asc文件,否则安装agent的时候会报以下错误:

9.对于离线安装CDH6.x,分发Parcel出现hash校验失败的问题,是因为在CM6中修复了一个bug,让它不再忽略由http服务器发送的“Content-Encoding”的header信息,但是我们在Redhat中安装的httpd服务,当它传输parcel文件时,默认会错误的设置“Content-Encoding”。于是CM server会错误的认为parcel文件已经被httpd压缩并尝试解压缩。所以会导致失败。解决办法是参考2.8章节的,设置httpd的conf文件,AddType application/x-gzip .gz .tgz .parcel,然后重启httpd服务和CM服务。这个问题在beta的时候就已经存在了,具体请参考《0350-Redhat7.4安装CDH6.0_beta1时分发Parcel异常分析

10.与CDH6.1的安装相比,6.2/6.3有一些细微的差别,首先多了一个步骤可以自定义集群的名字,默认“Cluster 1”;另外在“检查主机”那个步骤,多了一个选项“检查网络性能”,这个功能是为CDH6.2/6.3的新功能SDX服务的,因为SDX是存储计算分离的架构,如果网络是瓶颈,可能并不适合SDX。

本文分享自微信公众号 - Hadoop实操(gh_c4c535955d0f)

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。

原始发表时间:2019-11-10

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏数据和云

如何巧妙处理enq: TX - index contention问题?

某客户生产系统核心库在2019-08-21 23:50:00左右出现了业务告警,应用无法连接的情况。

10110
来自专栏安全漏洞环境学习

Joomla 3.4.6-RCE漏洞复现

Joomla是一套全球有名的CMS系统,基于PHP语言加上MySQL数据库所开发出来的WEB软件系统,目前最新版本是3.9.12。可以在多种不同的平台上部署并且...

15910
来自专栏Java3y

网络协议常见面试题

答:Http协议运行在TCP之上,明文传输,客户端与服务器端都无法验证对方的身份;Https是身披SSL(Secure Socket Layer)外壳的Http...

15470
来自专栏网站漏洞修复

APP渗透测试基本内容与漏洞扫描介绍

天气越来越凉爽,在对客户网站代码进行渗透测试,漏洞测试的同时我们SINE安全渗透技术要对客户的网站源代码进行全方位的安全检测与审计,只有真正的了解网站,才能更好...

9540
来自专栏生信技能树

多个探针对应同一个基因取最大值的代码进化历史

最近全国巡讲的学员又问到了多个探针对应同一个基因取最大值类似的问题,我们的斯老师找到了我三年前的博客:多个探针对应一个基因,取平均值或者最大值 我看到里面的留言...

17540
来自专栏AustinDatabases

阴沟翻船之 MYSQL MHA 故障 SSH timeout 与 Binlog not found

MYSQL MHA 的安装估计很多地方都是自动化安装的了,流水线方式。个人安装的MHA 的集群虽然没有几百台,但基本上已经突破了三位数,按理说安装应该是不会出什...

11310
来自专栏爬蜥的学习之旅

以 合同 - 产品 - 收入确认 示例来讲解领域逻辑模式

根据合同类型的不同,会有不同的产品,产品的不同,有不同的收入方式。已知每个合同的合同ID,假设产品类型现有的收入确认方式为(s->a),(w->b),(d->c...

7020
来自专栏JAVA杂谈

使用Prometheus监控MySQL性能指标

这里使用--web.listen-address=":9200"修改默认启动端口,防止跟上文中启动的服务器监控冲突。

26140
来自专栏小麦苗的DB宝专栏

【DB笔试面试667】在Oracle中,贵公司的数据库有多大?大一点的表有多大?有多少行?

对于数据库的大小,需要注意的问题是数据库的大小不能以表空间的分配大小而论,而应该以表空间的占用空间大小而论,并且需要减掉SYSTEM、SYSAUX、TEMP和U...

11860
来自专栏吉林乌拉

Redis中的事务管理

在之前的文章中我们已经介绍了很多关于Redis中的命令,虽然命令的执行是原子性的,但是如果多条命令组合在一起则Redis就不能保证组合命令的原子性了。并且如果我...

8010

扫码关注云+社区

领取腾讯云代金券

年度创作总结 领取年终奖励