CentOS7下利用Ambari搭建HDP大数据平台

yuanfan2012

发布于 2020-09-01 15:06:35

3K0

发布于 2020-09-01 15:06:35

Ambari介绍

Apache Ambari是一个基于Web的支持Apache Hadoop集群的供应、管理和监控的开源工具， Ambari已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI进行可视化的集群管理，简化了大数据平台的安装、使用难度。

Ambari体系结构

Ambari 自身也是一个分布式架构的软件，主要由两部分组成：Ambari Server 和 Ambari Agent。

简单来说，用户通过Ambari Server通知 Ambari Agent 安装对应的软件；Agent 会定时地发送各个机器每个软件模块的状态给 Ambari Server，最终这些状态信息会呈现在 Ambari 的 GUI，方便用户了解到集群的各种状态，并进行相应的维护。Ambari Server 从整个集群上收集信息。每个主机上都有 Ambari Agent, Ambari Server 通过 Ambari Agent 控制每部主机。

下面介绍CentOS7系统下利用Ambari搭建HDP大数据平台

参考官方文档

https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation/content/install-ambari-server-rhel7.html https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation/content/hdp_26_repositories.html https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation/content/ambari_repositories.html

一、准备条件

1、jdk安装包 jdk-8u261-linux-x64.tar.gz

2、提前下载好ambari,HDP,HDP-UTILS 本地仓库Tarball文件

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari-2.6.2.2-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

二、操作系统基础环境准备

先在ambari机器上配置JDK，准备基础环境

1)、关闭防火墙

systemctl disable firewalld
systemctl stop firewalld

2)、关闭SELINUX

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

3)、设置ulimit 文件打开数为10000

在/etc/profile最后一行添加ulimit -n 10000

4)、安装JDK，配置环境变量

cd /opt
tar -zxvf jdk-8u261-linux-x64.tar.gz
mv jdk1.8.0_261 /usr/local/
cd /usr/local/
mv jdk1.8.0_261 jdk

#在/etc/profile中添加如下行
vi /etc/profile
export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/lib
PATH=$PATH:$JAVA_HOME/bin

5)、source /etc/profile生效

最后检查java，ulimit值

以上配置做好之后，可以poweroff关机做一个虚拟机快照，按这个虚拟机模板克隆4台机器分别为bigdata1,bigdata2,bigdata3,bigdata4

三、主机名修改与免密钥登录配置

1)5台主机分别修改主机名

hostnamectl set-hostname ambari
hostnamectl set-hostname bigdata1
hostnamectl set-hostname bigdata2
hostnamectl set-hostname bigdata3
hostnamectl set-hostname bigdata4

2)配置ambari-server免密钥登录其他4台节点

ssh-keygen

3)配置主机域名解析

vi /etc/hosts
添加如下行
10.20.90.59 ambari
10.20.90.60 bigdata1
10.20.90.61 bigdata2
10.20.90.62 bigdata3
10.20.90.63 bigdata4

4)拷贝公钥

ssh-copy-id root@bigdata1
ssh-copy-id root@bigdata2
ssh-copy-id root@bigdata3
ssh-copy-id root@bigdata4
ssh-copy-id root@ambari

5)主机域名解析文件拷贝到4台节点上

scp /etc/hosts root@bigdata1:/etc
scp /etc/hosts root@bigdata2:/etc
scp /etc/hosts root@bigdata3:/etc
scp /etc/hosts root@bigdata4:/etc

并验证可以正常免密登录

四、配置Ambari HDP以及HDP-UTILS 本地yum源

在ambari服务器上配置Ambari HDP以及HDP-UTILS 本地yum源

yum install -y httpd
cd /opt
mkdir /var/www/html/ambari
mkdir /var/www/html/hdp
tar -zxf ambari-2.6.2.2-centos7.tar.gz -C /var/www/html/ambari/
tar -zxf HDP-2.6.5.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
tar -zxf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp/
 
systemctl start httpd
systemctl enable httpd

SFTP上传到/opt目录下

验证Ambari HDP以及HDP-UTILS 本地yum源均可以正常访问

http://10.20.90.59/ambari/ambari/centos7/2.6.2.2-1/
http://10.20.90.59/hdp/HDP/centos7/2.6.5.0-292/
http://10.20.90.59/hdp/HDP-UTILS/centos7/1.1.0.22/

针对下面这个错误的解决办法

删除或者移走html文件

配置ambari YUM源

cat >/etc/yum.repos.d/ambari.repo<<EOF
#VERSION_NUMBER=2.6.2.2-1
[Ambari-2.6.2.2-1]
name=2.6.2.2-1 - Updates
baseurl=http://10.20.90.59/ambari/ambari/centos7/2.6.2.2-1/
gpgcheck=1
gpgkey=http://10.20.90.59/ambari/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF

#配置hdp YUM源

cat >/etc/yum.repos.d/hdp.repo<<EOF
#VERSION_NUMBER=2.6.5.0-292
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0-292
baseurl=http://10.20.90.59/hdp/HDP/centos7/2.6.5.0-292/
gpgcheck=1
gpgkey=http://10.20.90.59/hdp/HDP/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://10.20.90.59/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=1
gpgkey=http://10.20.90.59/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF

cat /etc/yum.repos.d/ambari.repo
cat /etc/yum.repos.d/hdp.repo

五、安装ambari-server

yum -y install ambari-server

六、安装MariaDB数据库并配置数据库

1、安装MariaDB数据库

yum -y install mariadb-server
systemctl enable mariadb.service
systemctl start mariadb.service
netstat -anp | grep 3306

2、设置数据库初始密码

mysqladmin -uroot password "ambari"

3、设置可以远程登录

mysql -uroot -pambari
grant all privileges on *.* to 'root'@'ambari' identified by 'ambari' with grant option;
grant all privileges on *.* to 'root'@'10.20.90.59' identified by 'ambari' with grant option;
flush privileges;
systemctl restart mariadb.service

4、创建ambari数据库

mysql -uroot -pambari -hambari 
create database ambari DEFAULT CHARSET utf8;
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

七、ambari-server初始化配置

[root@ambari home]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): 
Port (3306): 
Database name (ambari): 
Username (ambari): root
Enter Database Password (bigdata): ambari
Re-enter password: 
Configuring ambari database...
WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties.
Press <enter> to continue.
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
....ambari-admin-2.6.2.2.1.jar
.......
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

拷贝jdbc驱动包，并在ambari.properties配置文件中添加jdbc驱动的配置

mkdir /usr/share/java
cp /home/mysql-connector-java-8.0.20.jar /usr/share/java/mysql-connector-java.jar
vi /etc/ambari-server/conf/ambari.properties 
#追加一行
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

ambari-server start

八、ambari web界面向导搭建集群

注意：这里的URL最后面的/斜杠需要去掉，（图误）

九、排错

1、Confirm Hosts时出现安装ambari-agent的报错的解决方法

修改 /etc/ambari-agent/conf/ambari-agent.ini 文件，在 [security] 配置部分添加
[security]
...
force_https_protocol=PROTOCOL_TLSv1_2

2、yum.conf配置文件修改

vi /etc/yum.conf 
installonly_limit=5改为
installonly_limit=600

本文参与腾讯云自媒体分享计划，分享自微信公众号。

原始发表：2020-08-27，如有侵权请联系 cloudcommunity@tencent.com 删除

yum

本文分享自 WalkingCloud 微信公众号，前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体分享计划，欢迎热爱写作的你一起参与！

登录后参与评论

0 条评论

热度