大数据源码编译系列——Ambari2.5.1编译

  |   0 评论   |   3,604 浏览

准备环境

JDK8

Maven

Ant

Git

GCC

Cmake

安装NodeJS

安装brunch

Python setuptools 

[root@nn1 src]# wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg#md5=bfa92100bd772d5a213eedd356d64086 
[root@nn1 src]# sh setuptools-0.6c11-py2.6.egg

其它

[root@nn1 src]# yum install rpm-build gcc make gcc-c++ openssl-devel wget python-devel

准备安装包

下载地址

[root@nn1 src]# wget http://mirror.bit.edu.cn/apache/ambari/ambari-2.5.1/apache-ambari-2.5.1-src.tar.gz
[root@nn1 src]# tar zxvf apache-ambari-2.5.1-src.tar.gz -C /usr/local/src/
[root@nn1 src]# cd apache-ambari-2.5.1-src
[root@nn1 apache-ambari-2.5.1-src]# mvn versions:set -DnewVersion=2.5.1.0.0
[root@nn1 apache-ambari-2.5.1-src]# pushd ambari-metrics
[root@nn1 apache-ambari-2.5.1-src]# mvn versions:set -DnewVersion=2.5.1.0.0
[root@nn1 apache-ambari-2.5.1-src]# popd

编译

[root@nn1 apache-ambari-2.5.1-src]# mvn -B clean install package rpm:rpm -DnewVersion=2.5.1.0.0 -DskipTests -Dpython.ver="python >= 2.6"

编译结果

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Ambari Main ........................................ SUCCESS [ 47.883 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [  0.563 s]
[INFO] Ambari Web ......................................... SUCCESS [02:00 min]
[INFO] Ambari Views ....................................... SUCCESS [  5.695 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 44.663 s]
[INFO] utility ............................................ SUCCESS [  1.338 s]
[INFO] ambari-metrics ..................................... SUCCESS [  3.368 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [ 20.589 s]
[INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [ 19.698 s]
[INFO] Ambari Metrics Flume Sink .......................... SUCCESS [ 12.077 s]
[INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [  7.148 s]
[INFO] Ambari Metrics Storm Sink .......................... SUCCESS [  9.242 s]
[INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [  7.495 s]
[INFO] Ambari Metrics Collector ........................... SUCCESS [05:59 min]
[INFO] Ambari Metrics Monitor ............................. SUCCESS [  6.700 s]
[INFO] Ambari Metrics Grafana ............................. SUCCESS [  3.747 s]
[INFO] Ambari Metrics Assembly ............................ SUCCESS [06:58 min]
[INFO] Ambari Server ...................................... SUCCESS [27:48 min]
[INFO] Ambari Functional Tests ............................ SUCCESS [ 59.642 s]
[INFO] Ambari Agent ....................................... SUCCESS [07:22 min]
[INFO] Ambari Client ...................................... SUCCESS [  1.510 s]
[INFO] Ambari Python Client ............................... SUCCESS [  8.236 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [ 17.471 s]
[INFO] Ambari Shell ....................................... SUCCESS [  1.136 s]
[INFO] Ambari Python Shell ................................ SUCCESS [  3.367 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [ 23.759 s]
[INFO] ambari-logsearch ................................... SUCCESS [  1.902 s]
[INFO] Ambari Logsearch Appender .......................... SUCCESS [  2.372 s]
[INFO] Ambari Logsearch Portal ............................ SUCCESS [14:31 min]
[INFO] Ambari Logsearch Log Feeder ........................ SUCCESS [ 54.101 s]
[INFO] Ambari Logsearch Solr Client ....................... SUCCESS [ 18.693 s]
[INFO] Ambari Infra Solr Plugin ........................... SUCCESS [10:41 min]
[INFO] Ambari Logsearch Assembly .......................... SUCCESS [  4.853 s]
[INFO] Ambari Logsearch Integration Test .................. SUCCESS [05:34 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:27 h
[INFO] Finished at: 2017-06-27T20:44:02-07:00
[INFO] Final Memory: 294M/1001M
[INFO] ------------------------------------------------------------------------

查看编译生成的文件

[root@nn1 apache-ambari-2.5.1-src]# ll ambari-server/target/rpm/ambari-server/RPMS/x86_64/
total 99744
-rw-r--r-- 1 root root 102131412 Jun 27 20:02 ambari-server-2.5.1.0-0.x86_64.rpm
[root@nn1 apache-ambari-2.5.1-src]# ll ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/
total 37504
-rw-r--r-- 1 root root 38402868 Jun 27 20:10 ambari-agent-2.5.1.0-0.x86_64.rpm

可能遇到的错误

storm-core版本不对

错误信息

[ERROR] Failed to execute goal on project ambari-metrics-storm-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink:jar:2.5.1.0.0: Could not find artifact org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT in apache-hadoop (http://repo.hortonworks.com/content/groups/public/) -> [Help 1]

解决方法

[root@nn1 apache-ambari-2.5.1-src]# sed -i "s/<storm.version>1.1.0-SNAPSHOT</<storm.version>1.1.0</g" `grep storm -rl ambari-metrics/ambari-metrics-storm-sink/pom.xml`

没有psutil

报错信息

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (psutils-compile) on project ambari-metrics-host-monitoring: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="/usr/local/src/apache-ambari-2.5.1-src/ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil" executable="/usr/local/src/apache-ambari-2.5.1-src/ambari-metrics/ambari-metrics-host-monitoring/../../ambari-common/src/main/unix/ambari-python-wrap">... @ 4:293 in /usr/local/src/apache-ambari-2.5.1-src/ambari-metrics/ambari-metrics-host-monitoring/target/antrun/build-psutils-compile.xml

缺少python  psutils模块

解决办法

[root@nn1 src]# wget https://pypi.python.org/packages/source/p/psutil/psutil-2.0.0.tar.gz
[root@nn1 src]# tar -xzvf psutil-2.0.0.tar.gz
[root@nn1 src]# cd psutil-2.0.0
[root@nn1 psutil-2.0.0]# python setup.py install

error: Python.h: No such file or directory

安装python setup.py install时出现如下错误

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.6 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.6/psutil/_psutil_linux.o
psutil/_psutil_linux.c:12:20: error: Python.h: No such file or directory
In file included from psutil/_psutil_linux.c:23:
psutil/_psutil_linux.h:11: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token

解决办法

[root@nn1 psutil-2.0.0]# yum -y install python-devel

Connect to github-production-release-asset-2e65be.s3.amazonaws.com:443

报错信息

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:install-node-and-yarn (install node and yarn) on project ambari-web: Could not download Yarn: Could not download https://github.com/yarnpkg/yarn/releases/download/v0.23.2/yarn-v0.23.2.tar.gz: Connect to github-production-release-asset-2e65be.s3.amazonaws.com:443 [github-production-release-asset-2e65be.s3.amazonaws.com/52.216.81.80] failed: Connection timed out -> [Help 1]

解决办法

下载

https://github.com/yarnpkg/yarn/releases/download/v0.23.2/yarn-v0.23.2.tar.gz

并上传至

~/.m2/repository/com/github/eirslett/yarn/0.23.2/yarn-0.23.2./

下载tar.gz包失败

使用迅雷下载相关包至本地,并放置在/usr/local/ftp/目录下

[root@nn1 apache-ambari-2.5.1-src]# ll /usr/local/ftp/
total 806864
-rw-r--r--. 1 root root  23667150 Feb 15 06:59 grafana-2.6.0.linux-x64.tar.gz
-rw-r--r--. 1 root root 162765879 Feb 15 07:00 hadoop-2.6.0.2.2.4.2-0002.winpkg.zip
-rw-r--r--. 1 root root 210185315 Feb 15 06:59 hadoop-2.7.1.2.3.4.0-3347.tar.gz
-rw-r--r--. 1 root root  76388954 Feb 15 06:59 hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip
-rw-r--r--. 1 root root 101551599 Feb 15 07:00 hbase-1.1.2.2.3.4.0-3347.tar.gz
-rw-r--r--. 1 root root 251655846 Feb 15 06:59 phoenix-4.4.0.2.3.4.0-3347.tar.gz


替换原链接为本地目录

[root@nn1 apache-ambari-2.5.1-src]# vim ambari-metrics/pom.xml

原内容

<hbase.tar>http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0-3347/tars/hbase-1.1.2.2.3.4.0-3347.tar.gz</hbase.tar>
<hbase.folder>hbase-1.1.2.2.3.4.0-3347</hbase.folder>
<hadoop.tar>http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0-3347/tars/hadoop-2.7.1.2.3.4.0-3347.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-2.7.1.2.3.4.0-3347</hadoop.folder>
<hbase.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip</hbase.winpkg.zip>
<hbase.winpkg.folder>hbase-0.98.4.2.2.4.2-0002-hadoop2</hbase.winpkg.folder>
<hadoop.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip</hadoop.winpkg.zip>
<hadoop.winpkg.folder>hadoop-2.6.0.2.2.4.2-0002</hadoop.winpkg.folder>
<grafana.folder>grafana-2.6.0</grafana.folder>
<grafana.tar>https://grafanarel.s3.amazonaws.com/builds/grafana-2.6.0.linux-x64.tar.gz</grafana.tar>
<phoenix.tar>http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0-3347/tars/phoenix-4.4.0.2.3.4.0-3347.tar.gz</phoenix.tar>

替换后

<hbase.tar>file:///usr/local/ftp/hbase-1.1.2.2.3.4.0-3347.tar.gz</hbase.tar>
<hbase.folder>hbase-1.1.2.2.3.4.0-3347</hbase.folder>
<hadoop.tar>file:///usr/local/ftp/hadoop-2.7.1.2.3.4.0-3347.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-2.7.1.2.3.4.0-3347</hadoop.folder>
<hbase.winpkg.zip>file:///usr/local/ftp/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip</hbase.winpkg.zip>
<hbase.winpkg.folder>hbase-0.98.4.2.2.4.2-0002-hadoop2</hbase.winpkg.folder>
<hadoop.winpkg.zip>file:///usr/local/ftp/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip</hadoop.winpkg.zip>
<hadoop.winpkg.folder>hadoop-2.6.0.2.2.4.2-0002</hadoop.winpkg.folder>
<grafana.folder>grafana-2.6.0</grafana.folder>
<grafana.tar>file:///usr/local/ftp/grafana-2.6.0.linux-x64.tar.gz</grafana.tar>
<phoenix.tar>file:///usr/local/ftp/phoenix-4.4.0.2.3.4.0-3347.tar.gz</phoenix.tar>

效果图

image.png


读后有收获可以支付宝请作者喝咖啡