Ambari2.4 源码编译

程裕强

发布于 2022-05-06 10:42:40

4990

发布于 2022-05-06 10:42:40

文章被收录于专栏：大数据学习笔记

编译Ambari

第一步：Ambari编译前环境准备

参考网址：https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development

1. JDK

1.1 卸载原有JDK

[root@node~]# rpm -qa|grep jdk java-1.6.0-openjdk-devel-1.6.0.35-1.13.7.1.el6_6.x86_64 java-1.6.0-openjdk-1.6.0.35-1.13.7.1.el6_6.x86_64 [root@node~]# yum remove -y java-1.6.0-openjdk [root@node~]# rpm -qa|grep java tzdata-java-2015e-1.el6.noarch java_cup-0.10k-5.el6.x86_64 java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 [root@node~]# yum remove -y java

1.2 安装Oracle官方jdk7

[root@node~]# scp jdk-7u67-linux-x64.gz 172.16.54.166:/opt

1.3 配置环境JDK变量

2.安装Maven

2.1 下载http://maven.apache.org/

[root@node~]# tar -zxvf apache-maven-3.0.5-bin.tar.gz

2.2 配置环境变量

[root@node apache-maven-3.0.5]# vi /etc/profile export JAVA_HOME=/opt/jdk1.7.0_67 export MAVEN_HOME=/opt/apache-maven-3.0.5 export PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true" [root@node apache-maven-3.0.5]# source /etc/profile

2.3 测试Maven

[root@node~ apache-maven-3.0.5]# mvn -v Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800) Maven home: /opt/apache-maven-3.0.5 Java version: 1.7.0_67, vendor: Oracle Corporation Java home: /opt/jdk1.7.0_67/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-573.el6.x86_64", arch: "amd64", family: "unix"

2.4 更换国内源

[root@node apache-maven-3.0.5]# cd conf [root@node conf]# vi settings.xml  <mirror> <id>alimaven</id> <mirrorOf>central</mirrorOf> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/repositories/central/</url> </mirror>

3.Python

3.1 检测python版本

[root@cos6 opt]# python Python 2.6.6 (r266:84292, Jul 23 2015, 15:22:56) [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> exit();

3.2 setuptools

[root@east81 Downloads]# scp setuptools-0.6c11-py2.7.egg 172.16.54.166:/opt [root@vnode1 opt]# sh setuptools-0.6c11-py2.7.egg Processing setuptools-0.6c11-py2.7.egg Removing /usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg Copying setuptools-0.6c11-py2.7.egg to /usr/lib/python2.7/site-packages setuptools 0.6c11 is already the active version in easy-install.pth Installing easy_install script to /usr/bin Installing easy_install-2.7 script to /usr/bin Installed /usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg Processing dependencies for setuptools==0.6c11 Finished processing dependencies for setuptools==0.6c11

4.安装rpmbuild

4.1检测rpmbuild

[root@vnode1 opt]# rpm -qa | grep rpm-build

4.2安装rpmbuild

[root@vnode1 opt]# yum install -y rpm-build

5.安装gcc-c++

[root@vnode1 app]# yum install -y gcc-c++ [root@vnode0 ~]# yum install -y gcc make gcc-c++ openssl-devel wget

6.安装nodejs

6.1 下载nodejs

https://nodejs.org/download/release/v0.10.44/node-v0.10.44-linux-x64.tar.gz [root@cos6 opt]# tar -zxvf node-v0.10.44-linux-x64.tar.gz

6.2 配置nodejs环境变量

[root@cos6 opt]# cd node-v0.10.44-linux-x64 [root@cos6 node-v0.10.44-linux-x64]# ls bin ChangeLog include lib LICENSE README.md share [root@cos6 node-v0.10.44-linux-x64]# pwd /opt/node-v0.10.44-linux-x64 [root@cos6 node-v0.10.44-linux-x64]# vi /etc/profile #nodejs export NODE_HOME=/opt/node-v0.10.44-linux-x64 export PATH=$PATH:$NODE_HOME/bin 注意NODE_HOME前不要少了$ [root@cos6 node-v0.10.44-linux-x64]# source /etc/profile

6.3 检测nodejs命令

[root@vnode1 ~]# node -v v0.10.44 [root@vnode1 ~]# npm -v 2.15.0

7 安装brunch

7.1 查找node根目录

[root@vnode1 opt]# npm root -g /opt/node-v0.10.44-linux-x64/lib/node_modules

[root@vnode1 opt]# cd /opt/node-v0.10.44-linux-x64/lib/node_modules

7.2 更改taobao源

[root@vnode1 node_modules]# npm config set registry https://registry.npm.taobao.org

[root@vnode1 node_modules]# npm info underscore

7.3 安装1.7.20版本的brunch

[root@vnode1 node_modules]# npm install -g brunch@1.7.20 npm WARN deprecated minimatch@1.0.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue npm WARN optional dep failed, continuing fsevents@0.3.8 npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree. /opt/node-v0.10.44-linux-x64/bin/brunch -> /opt/node-v0.10.44-linux-x64/lib/node_modules/brunch/bin/brunch > brunch@1.7.20 postinstall /opt/node-v0.10.44-linux-x64/lib/node_modules/brunch > node setup.js postinstall brunch@1.7.20 /opt/node-v0.10.44-linux-x64/lib/node_modules/brunch ├── debug@0.7.4 ├── async-waterfall@0.1.5 ├── commonjs-require-definition@0.1.2 ├── async-each@0.1.6 ├── commander@2.0.0 ├── ncp@0.4.2 ├── mkdirp@0.3.5 ├── init-skeleton@0.2.4 (rimraf@2.2.8) ├── anymatch@1.0.0 (minimatch@1.0.0) ├── read-components@0.6.1 (component-builder@0.10.1) ├── source-map@0.1.43 (amdefine@1.0.1) ├── coffee-script@1.8.0 ├── chokidar@0.12.6 (readdirp@1.3.0) ├── anysort@1.0.1 (anymatch@1.3.0) ├── pushserve@0.1.6 (connect-slashes@0.0.11, express@3.3.8) └── loggy@0.2.2 (growl@1.8.1, ansi-color@0.2.1, date-utils@1.2.21) [root@vnode1 node_modules]#

7.4 可能存在的问题

安装失败时，卸载brunch [root@cos6 conf]# npm remove -g brunch [root@vnode1 apache-ambari-2.4.2-src]# npm install node-gyp node-gyp rebuilt卡住 > contextify@0.1.15 install /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/contextify > node-gyp rebuild

8 海量文件设置

可能报错：too many files are opened

[root@cos6 node_modules]# echo fs.inotify.max_user_watches=524288 |tee -a /etc/sysctl.conf && sysctl -p

fs.inotify.max_user_watches=524288 net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 fs.inotify.max_user_watches = 524288 将每个进程可以打开的文件数目加大到10000，缺省为1024 [root@vnode1 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimite pending signals (-i) 7217 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimite open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7217 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@vnode1 ~]# ulimit -n 10000

9 安装git

[root@cos6 ~]# git –version -bash: git: command not found [root@cos6 setuptools-32.1.2]# cd

[root@cos6 ~]# yum install -y git

10 安装ant

[root@vnode1 apache-ambari-2.4.2-src]# yum install -y ant

第二步：编译Ambari源码

1、下载ambari源码

wget http://www.apache.org/dist/mbari/ambari-2.4.2/apache-ambari-2.4.2-src.tar.gz [root@vnode1 opt]# tar -zxvf apache-ambari-2.4.2-src.tar.gz

2、给Ambari打上版本号

[root@vnode1 apache-ambari-2.4.2-src]# mvn versions:set -DnewVersion=2.4.2.0 [root@vnode1 apache-ambari-2.4.2-src]# pushd ambari-metrics /opt/apache-ambari-2.4.2-src/ambari-metrics /opt/apache-ambari-2.4.2-src 注意，返回上一级目录 [root@vnode1 ambari-metrics]# cd .. [root@vnode1 apache-ambari-2.4.2-src]# mvn versions:set -DnewVersion=2.4.2.0 [root@vnode1 apache-ambari-2.4.2-src]# popd /opt/apache-ambari-2.4.2-src

3、开始编译

[root@vnode1 apache-ambari-2.4.2-src]# mvn -B clean install package rpm:rpm -DnewVersion=2.4.2.0 -DskipTests -Dpython.ver="python >= 2.6"

.................

[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ....................................... SUCCESS [4.304s] [INFO] Apache Ambari Project POM ......................... SUCCESS [0.149s] [INFO] Ambari Web ........................................ SUCCESS [1:21.863s] [INFO] Ambari Views ...................................... SUCCESS [1.396s] [INFO] Ambari Admin View ................................. SUCCESS [6:00.715s] [INFO] ambari-metrics .................................... SUCCESS [0.454s] [INFO] Ambari Metrics Common ............................. SUCCESS [1.182s] [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [1.961s] [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [1.042s] [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [0.705s] [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [2.377s] [INFO] Ambari Metrics Storm Sink (Legacy) ................ SUCCESS [1.887s] [INFO] Ambari Metrics Collector .......................... SUCCESS [1:12:45.829s] [INFO] Ambari Metrics Monitor ............................ SUCCESS [2.403s] [INFO] Ambari Metrics Grafana ............................ SUCCESS [28.278s] [INFO] Ambari Metrics Assembly ........................... SUCCESS [35:59.608s] [INFO] Ambari Server ..................................... SUCCESS [4:29.713s] [INFO] Ambari Functional Tests ........................... SUCCESS [15.422s] [INFO] Ambari Agent ...................................... SUCCESS [52.274s] [INFO] Ambari Client ..................................... SUCCESS [0.175s] [INFO] Ambari Python Client .............................. SUCCESS [1.595s] [INFO] Ambari Groovy Client .............................. SUCCESS [17.700s] [INFO] Ambari Shell ...................................... SUCCESS [0.233s] [INFO] Ambari Python Shell ............................... SUCCESS [1.408s] [INFO] Ambari Groovy Shell ............................... SUCCESS [15.462s] [INFO] ambari-logsearch .................................. SUCCESS [0.218s] [INFO] Ambari Logsearch Appender ......................... SUCCESS [0.431s] [INFO] Ambari Logsearch Solr Client ...................... SUCCESS [1.946s] [INFO] Ambari Logsearch Portal ........................... SUCCESS [2:14.732s] [INFO] Ambari Logsearch Log Feeder ....................... SUCCESS [8.750s] [INFO] Ambari Logsearch Assembly ......................... SUCCESS [0.328s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:05:35.686s [INFO] Finished at: Fri Dec 23 07:38:57 EST 2016 [INFO] Final Memory: 365M/1295M [INFO] ------------------------------------------------------------------------ [root@vnode1 apache-ambari-2.4.2-src]#

4、编译成功后，安装rpm包

Ambari安装需要ambari-server安装到管理机上，ambari-agent安装到集群节点上。 Ambari-server的rpm包 ambari-server/target/rpm/ambari-server/RPMS/noarch/ambari-server-*.noarch.rpm /opt/apache-ambari-2.4.2-src/ambari-server/target/rpm/ambari-server/RPMS/x86_64/ambari-server-2.4.2.0-0.x86_64.rpm Ambari-agent的rpm包 ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-*.rpm /opt/apache-ambari-2.4.2-src/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-2.4.2.0-0.x86_64.rpm

第三、存在问题

1、NodeJS版本不对 [root@vnode1 opt]# node -v v6.9.1 可能通过rpm包安装过高版本的node 尝试卸载： [root@vnode1 opt]# yum remove -y nodejs Loaded plugins: fastestmirror Resolving Dependencies --> Running transaction check ---> Package nodejs.x86_64 1:6.9.1-4.el7 will be erased --> Processing Dependency: nodejs = 1:6.9.1-4.el7 for package: 1:npm-3.10.8-1.6.9.1.4.el7.x86_64 --> Running transaction check ---> Package npm.x86_64 1:3.10.8-1.6.9.1.4.el7 will be erased --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================= Removing: nodejs x86_64 1:6.9.1-4.el7 @epel 16 M Removing for dependencies: npm x86_64 1:3.10.8-1.6.9.1.4.el7 @epel 9.7 M Transaction Summary ================================================================================================================================================= Remove 1 Package (+1 Dependent package) Installed size: 26 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Erasing : 1:npm-3.10.8-1.6.9.1.4.el7.x86_64 1/2 Erasing : 1:nodejs-6.9.1-4.el7.x86_64 2/2 Verifying : 1:nodejs-6.9.1-4.el7.x86_64 1/2 Verifying : 1:npm-3.10.8-1.6.9.1.4.el7.x86_64 2/2 Removed: nodejs.x86_64 1:6.9.1-4.el7 Dependency Removed: npm.x86_64 1:3.10.8-1.6.9.1.4.el7 Complete! 看来之前真的安装过高版本的node 再次解压缩，配置环境变量 [root@vnode1 opt]# source /etc/profile [root@vnode1 opt]# node -v v0.10.44 2、下载phantomjs-1.9.8-linux-x86_64.tar.bz2卡住 > phantomjs@1.9.20 install /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs > node install.js PhantomJS not found on PATH Downloading https://github.com/Medium/phantomjs/releases/download/v1.9.19/phantomjs-1.9.8-linux-x86_64.tar.bz2 Saving to /opt/apache-ambari-2.2.2-src/ambari-web/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2 Receiving... 单独下载https://github.com/Medium/phantomjs/releases/download/v1.9.19/phantomjs-1.9.8-linux-x86_64.tar.bz2 [root@east81 Public]# scp phantomjs-1.9.8-linux-x86_64.tar.bz2 172.16.54.131:/opt [root@vnode1 opt]# mkdir -p /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs/phantomjs/ [root@vnode1 opt]# cp phantomjs-1.9.8-linux-x86_64.tar.bz2 /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs/phantomjs/phantomjs-1.9.8-linux-x86_64.tar.bz2 再次确认该文件已经复制过去： [root@vnode1 opt]# ls /opt/apache-ambari-2.4.2-src/ambari-web/node_modules/phantomjs/phantomjs phantomjs-1.9.8-linux-x86_64.tar.bz2 //npm install phantomjs -g 去掉clean，继续编译： [root@vnode1 apache-ambari-2.4.2-src]# mvn -B install package rpm:rpm -DnewVersion=2.4.2.0 -DskipTests -Dpython.ver="python >= 2.6" -X 3、下载node-v0.10.44-linux-x64.tar.gz出错 [INFO] Downloading Node.js from http://nodejs.org/dist/v0.10.44/node-v0.10.44-linux-x64.tar.gz to /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz [INFO] No proxy was configured, downloading directly [INFO] Extracting Node.js files in node_tmp [INFO] Unpacking /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz into /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ....................................... SUCCESS [4:04.773s] [INFO] Apache Ambari Project POM ......................... SUCCESS [7.184s] [INFO] Ambari Web ........................................ SUCCESS [2:33.179s] [INFO] Ambari Views ...................................... SUCCESS [4:35.686s] [INFO] Ambari Admin View ................................. FAILURE [9:32.324s] [INFO] ambari-metrics .................................... SKIPPED [INFO] Ambari Metrics Common ............................. SKIPPED [INFO] Ambari Metrics Hadoop Sink ........................ SKIPPED [INFO] Ambari Metrics Flume Sink ......................... SKIPPED [INFO] Ambari Metrics Kafka Sink ......................... SKIPPED [INFO] Ambari Metrics Storm Sink ......................... SKIPPED [INFO] Ambari Metrics Collector .......................... SKIPPED [INFO] Ambari Metrics Monitor ............................ SKIPPED [INFO] Ambari Metrics Grafana ............................ SKIPPED [INFO] Ambari Metrics Assembly ........................... SKIPPED [INFO] Ambari Server ..................................... SKIPPED [INFO] Ambari Agent ...................................... SKIPPED [INFO] Ambari Client ..................................... SKIPPED [INFO] Ambari Python Client .............................. SKIPPED [INFO] Ambari Groovy Client .............................. SKIPPED [INFO] Ambari Shell ...................................... SKIPPED [INFO] Ambari Python Shell ............................... SKIPPED [INFO] Ambari Groovy Shell ............................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:54.669s [INFO] Finished at: Wed Dec 21 14:20:25 CST 2016 [INFO] Final Memory: 24M/61M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm (install node and npm) on project ambari-admin: Could not extract the Node archive: Could not extract archive: '/opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz': EOFException -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :ambari-admin [root@cos6 apache-ambari-2.2.2-src]# 确认一下 [root@cos6 apache-ambari-2.2.2-src]# tar -zxvf /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz node-v0.10.44-linux-x64/ node-v0.10.44-linux-x64/bin/ node-v0.10.44-linux-x64/bin/npm node-v0.10.44-linux-x64/bin/node gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now 重新下载，通过迅雷下载http://nodejs.org/dist/v0.10.44/node-v0.10.44-linux-x64.tar.gz 然后复制到/opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz （node-v0.10.44-linux-x64.tar.gz更名为node.tar.gz） [root@cos6 opt]# cp node-v0.10.44-linux-x64.tar.gz /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz [root@cos6 opt]# tar -zxvf /opt/apache-ambari-2.2.2-src/ambari-admin/src/main/resources/ui/admin-web/node_tmp/node.tar.gz

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2016-12-21，如有侵权请联系 cloudcommunity@tencent.com 删除

apache