CentOS7下利用Ambari搭建HDP大数据平台
Ambari介绍
Apache Ambari是一个基于Web的支持Apache Hadoop集群的供应、管理和监控的开源工具， Ambari已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI进行可视化的集群管理，简化了大数据平台的安装、使用难度。
Ambari体系结构
Ambari 自身也是一个分布式架构的软件，主要由两部分组成：Ambari Server 和 Ambari Agent。
简单来说，用户通过Ambari Server通知 Ambari Agent 安装对应的软件；Agent 会定时地发送各个机器每个软件模块的状态给 Ambari Server，最终这些状态信息会呈现在 Ambari 的 GUI，方便用户了解到集群的各种状态，并进行相应的维护。Ambari Server 从整个集群上收集信息。每个主机上都有 Ambari Agent, Ambari Server 通过 Ambari Agent 控制每部主机。
下面介绍CentOS7系统下利用Ambari搭建HDP大数据平台
参考官方文档
https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation/content/install-ambari-server-rhel7.html https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation/content/hdp_26_repositories.html https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation/content/ambari_repositories.html
一、准备条件
1、jdk安装包 jdk-8u261-linux-x64.tar.gz
2、提前下载好ambari,HDP,HDP-UTILS 本地仓库Tarball文件
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari-2.6.2.2-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
二、操作系统基础环境准备
先在ambari机器上配置JDK，准备基础环境
1)、关闭防火墙
systemctl disable firewalld
systemctl stop firewalld
2)、关闭SELINUX
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
3)、设置ulimit 文件打开数为10000
在/etc/profile最后一行添加ulimit -n 10000
4)、安装JDK，配置环境变量
cd /opt
tar -zxvf jdk-8u261-linux-x64.tar.gz
mv jdk1.8.0_261 /usr/local/
cd /usr/local/
mv jdk1.8.0_261 jdk
#在/etc/profile中添加如下行
vi /etc/profile
export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/lib
PATH=$PATH:$JAVA_HOME/bin5)、source /etc/profile生效
最后检查java，ulimit值
以上配置做好之后，可以poweroff关机做一个虚拟机快照，按这个虚拟机模板克隆4台机器 分别为bigdata1,bigdata2,bigdata3,bigdata4
三、主机名修改与免密钥登录配置
1)5台主机分别修改主机名
hostnamectl set-hostname ambari
hostnamectl set-hostname bigdata1
hostnamectl set-hostname bigdata2
hostnamectl set-hostname bigdata3
hostnamectl set-hostname bigdata42)配置ambari-server免密钥登录其他4台节点
ssh-keygen
3)配置主机域名解析
vi /etc/hosts
添加如下行
10.20.90.59 ambari
10.20.90.60 bigdata1
10.20.90.61 bigdata2
10.20.90.62 bigdata3
10.20.90.63 bigdata44)拷贝公钥
ssh-copy-id root@bigdata1
ssh-copy-id root@bigdata2
ssh-copy-id root@bigdata3
ssh-copy-id root@bigdata4
ssh-copy-id root@ambari
5)主机域名解析文件拷贝到4台节点上
scp /etc/hosts root@bigdata1:/etc
scp /etc/hosts root@bigdata2:/etc
scp /etc/hosts root@bigdata3:/etc
scp /etc/hosts root@bigdata4:/etc
并验证可以正常免密登录
四、配置Ambari HDP以及HDP-UTILS 本地yum源
在ambari服务器上配置Ambari HDP以及HDP-UTILS 本地yum源
yum install -y httpd
cd /opt
mkdir /var/www/html/ambari
mkdir /var/www/html/hdp
tar -zxf ambari-2.6.2.2-centos7.tar.gz -C /var/www/html/ambari/
tar -zxf HDP-2.6.5.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
tar -zxf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp/
systemctl start httpd
systemctl enable httpd
SFTP上传到/opt目录下
验证Ambari HDP以及HDP-UTILS 本地yum源均可以正常访问
http://10.20.90.59/ambari/ambari/centos7/2.6.2.2-1/
http://10.20.90.59/hdp/HDP/centos7/2.6.5.0-292/
http://10.20.90.59/hdp/HDP-UTILS/centos7/1.1.0.22/
针对下面这个错误的解决办法
删除或者移走html文件
配置ambari YUM源
cat >/etc/yum.repos.d/ambari.repo<<EOF
#VERSION_NUMBER=2.6.2.2-1
[Ambari-2.6.2.2-1]
name=2.6.2.2-1 - Updates
baseurl=http://10.20.90.59/ambari/ambari/centos7/2.6.2.2-1/
gpgcheck=1
gpgkey=http://10.20.90.59/ambari/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
#配置hdp YUM源
cat >/etc/yum.repos.d/hdp.repo<<EOF
#VERSION_NUMBER=2.6.5.0-292
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0-292
baseurl=http://10.20.90.59/hdp/HDP/centos7/2.6.5.0-292/
gpgcheck=1
gpgkey=http://10.20.90.59/hdp/HDP/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://10.20.90.59/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=1
gpgkey=http://10.20.90.59/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
cat /etc/yum.repos.d/ambari.repo
cat /etc/yum.repos.d/hdp.repo五、安装ambari-server
yum -y install ambari-server
六、安装MariaDB数据库并配置数据库
1、安装MariaDB数据库
yum -y install mariadb-server
systemctl enable mariadb.service
systemctl start mariadb.service
netstat -anp | grep 3306
2、设置数据库初始密码
mysqladmin -uroot password "ambari"3、设置可以远程登录
mysql -uroot -pambari
grant all privileges on *.* to 'root'@'ambari' identified by 'ambari' with grant option;
grant all privileges on *.* to 'root'@'10.20.90.59' identified by 'ambari' with grant option;
flush privileges;
systemctl restart mariadb.service
4、创建ambari数据库
mysql -uroot -pambari -hambari
create database ambari DEFAULT CHARSET utf8;
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
七、ambari-server初始化配置
[root@ambari home]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari): root
Enter Database Password (bigdata): ambari
Re-enter password:
Configuring ambari database...
WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties.
Press <enter> to continue.
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
....ambari-admin-2.6.2.2.1.jar
.......
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.拷贝jdbc驱动包，并在ambari.properties配置文件中添加jdbc驱动的配置
mkdir /usr/share/java
cp /home/mysql-connector-java-8.0.20.jar /usr/share/java/mysql-connector-java.jar
vi /etc/ambari-server/conf/ambari.properties
#追加一行
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
ambari-server start
八、ambari web界面向导搭建集群
注意：这里的URL最后面的/斜杠需要去掉，（图误）
九、排错
1、Confirm Hosts时出现安装ambari-agent的报错的解决方法
修改 /etc/ambari-agent/conf/ambari-agent.ini 文件，在 [security] 配置部分添加
[security]
...
force_https_protocol=PROTOCOL_TLSv1_22、yum.conf配置文件修改
vi /etc/yum.conf
installonly_limit=5改为
installonly_limit=600举报
腾讯云开发者
