大数据源码编译系列——Kylin编译

  |   0 评论   |   4,451 浏览


准备环境

JDK8

Maven

Ant

Git

GCC

Cmake

准备文件

Kylin下载地址

[root@nn1 src]# cd /usr/local/src
[root@nn1 src]# wget http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.0.0/apache-kylin-2.0.0-src.tar.gz
[root@nn1 src]# tar zxvf apache-kylin-2.0.0-src.tar.gz -C /usr/local/src
[root@nn1 src]# cd apache-kylin-2.0.0

关闭DocLint特性

参考《关闭DocLint特性


否则会报以下错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.9.1:jar (attach-javadocs) on project kylin-core-common: MavenReportException: Error while creating archive: 
[ERROR] Exit code: 1 - /usr/local/src/apache-kylin-2.0.0/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java:152: warning: no @param for key
[ERROR]     final public void setProperty(String key, String value) {
[ERROR]                       ^
[ERROR] /usr/local/src/apache-kylin-2.0.0/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java:152: warning: no @param for value
[ERROR]     final public void setProperty(String key, String value) {
[ERROR]                       ^
[ERROR] /usr/local/src/apache-kylin-2.0.0/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java:468: warning: no @return
[ERROR]     public String getHiveUrl() {
[ERROR]                   ^
[ERROR] /usr/local/src/apache-kylin-2.0.0/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java:476: warning: no @return
[ERROR]     public String getHiveUser() {
[ERROR]                   ^
[ERROR] /usr/local/src/apache-kylin-2.0.0/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java:484: warning: no @return
[ERROR]     public String getHivePassword() {
[ERROR]                   ^
[ERROR] /usr/local/src/apache-kylin-2.0.0/core-common/src/main/java/org/apache/kylin/common/KylinVersion.java:116: error: self-closing element not allowed
[ERROR]      * <p/>
[ERROR]        ^
[ERROR] /usr/local/src/apache-kylin-2.0.0/core-common/src/main/java/org/apache/kylin/common/persistence/ResourceStore.java:127: warning: no @param for folderPath

修改Kylin依赖的JDK版本

[root@nn1 apache-kylin-2.0.0]# sed -i "s/javaVersion>1.7</javaVersion>1.8</g" `grep javaVersion -rl pom.xml`

否则会报以下错误

[WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed with message:
Detected JDK Version: 1.8.0-131 is not in the allowed range [1.7.0,1.7.1000}].

修改Kylin依赖的大数据组件版本

sed -i "s/2.6.0-cdh5.7.0/2.6.0-cdh5.11.1/g" `grep "cdh" -rl pom.xml`
sed -i "s/1.1.0-cdh5.7.0/1.1.0-cdh5.11.1/g" `grep "cdh" -rl pom.xml`
sed -i "s/1.2.0-cdh5.7.0/1.2.0-cdh5.11.1/g" `grep "cdh" -rl pom.xml`
sed -i "s/3.4.5-cdh5.7.0/3.4.5-cdh5.11.1/g" `grep "cdh" -rl pom.xml`
sed -i "s/cdh5.7/cdh5.11/g" `grep "cdh" -rl pom.xml`

安装NodeJS

详细步骤参考《安装NodeJS

编译

[root@nn1 apache-kylin-2.0.0]# build/script/package.sh -DskipTests -Pcdh5.11 -Papache-release -Dcheckstyle.skip

编译结果

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Kylin ....................................... SUCCESS [  7.379 s]
[INFO] Apache Kylin - Calcite Overrides ................... SUCCESS [  4.590 s]
[INFO] Apache Kylin - Core Common ......................... SUCCESS [  4.706 s]
[INFO] Apache Kylin - Core Metadata ....................... SUCCESS [  5.577 s]
[INFO] Apache Kylin - Core Dictionary ..................... SUCCESS [  2.960 s]
[INFO] Apache Kylin - Core Cube ........................... SUCCESS [  5.170 s]
[INFO] Apache Kylin - Core Job ............................ SUCCESS [  1.838 s]
[INFO] Apache Kylin - Core Storage ........................ SUCCESS [  1.498 s]
[INFO] Apache Kylin - MapReduce Engine .................... SUCCESS [ 10.200 s]
[INFO] Apache Kylin - HBase Storage ....................... SUCCESS [ 18.860 s]
[INFO] Apache Kylin - Spark Engine ........................ SUCCESS [ 17.612 s]
[INFO] Apache Kylin - Hive Source ......................... SUCCESS [01:21 min]
[INFO] Apache Kylin - Kafka Source ........................ SUCCESS [  5.811 s]
[INFO] Apache Kylin - Query ............................... SUCCESS [  1.356 s]
[INFO] Apache Kylin - Tool ................................ SUCCESS [  8.699 s]
[INFO] Apache Kylin - REST Server Base .................... SUCCESS [  9.900 s]
[INFO] Apache Kylin - REST Server ......................... SUCCESS [ 12.229 s]
[INFO] Apache Kylin - JDBC Driver ......................... SUCCESS [  4.932 s]
[INFO] Apache Kylin - Assembly ............................ SUCCESS [ 17.844 s]
[INFO] Apache Kylin - Tool Assembly ....................... SUCCESS [  4.252 s]
[INFO] Apache Kylin - Integration Test .................... SUCCESS [  5.533 s]
[INFO] Apache Kylin - Tomcat Extension .................... SUCCESS [  0.344 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:54 min
[INFO] Finished at: 2017-06-24T23:01:12-07:00
[INFO] Final Memory: 81M/555M
[INFO] ------------------------------------------------------------------------

查看生成的文件

[root@nn1 apache-kylin-2.0.0]# ll dist/
total 236968
-rw-r--r--. 1 root root 242653247 Jun 25 00:57 apache-kylin-2.0.0-bin.tar.gz

可能会报以下错误

cannot find symbol javax.annotation.Nullable

[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/src/apache-kylin-2.0.0/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/ZookeeperUtil.java:[23,23] error: cannot find symbol
[ERROR]   symbol:   class Nullable
  location: package javax.annotation
/usr/local/src/apache-kylin-2.0.0/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/ZookeeperUtil.java:[45,13] error: cannot find symbol
[ERROR]   symbol: class Nullable
/usr/local/src/apache-kylin-2.0.0/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/ZookeeperJobLock.java:[112,13] error: cannot find symbol

解决办法

vim pom.xml

<dependency>
  <groupId>com.google.code.findbugs</groupId>
  <artifactId>jsr305</artifactId>
  <version>3.0.2</version>
</dependency>

gpg: no default secret key: No secret key

[INFO] --- maven-gpg-plugin:1.5:sign (default) @ kylin ---
gpg: no default secret key: No secret key
gpg: signing failed: No secret key

解决办法

sed -i "s/<skip>false</<skip>true</g" `grep gpg -rl pom.xml`

here is 1 error reported by Checkstyle 6.19 with dev-support/checkstyle.xml ruleset.

解决办法

编译时加上

-Dcheckstyle.skip

MockHTable is not abstract and does not override abstract method getOperationTimeout() in Table

版本不兼容

[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/src/apache-kylin-2.0.0/server-base/src/main/java/org/apache/kylin/rest/security/MockHTable.java:[94,7] error: MockHTable is not abstract and does not override abstract method getRpcTimeout() in Table
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) on project kylin-server-base: Compilation failure
[ERROR] /usr/local/src/apache-kylin-2.0.0/server-base/src/main/java/org/apache/kylin/rest/security/MockHTable.java:[94,7] error: MockHTable is not abstract and does not override abstract method setRpcTimeout(int) in Table
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) on project kylin-server-base: Compilation failure
[ERROR] /usr/local/src/apache-kylin-2.0.0/server-base/src/main/java/org/apache/kylin/rest/security/MockHTable.java:[94,7] error: MockHTable is not abstract and does not override abstract method getOperationTimeout() in Table

解决办法

[root@nn1 apache-kylin-2.0.0]# vim /usr/local/src/apache-kylin-2.0.0/server-base/src/main/java/org/apache/kylin/rest/security/MockHTable.java

添加以下代码:

private int rpcTimeout = 3000;
@Override
public int getRpcTimeout() {
    //HConstants.HBASE_RPC_TIMEOUT_KEY
    //hbase.rpc.timeout
    return this.rpcTimeout;
}
public void setRpcTimeout(int rpcTimeout){
    this.rpcTimeout = rpcTimeout;
}
private int operationTimeout = 3000;
public void setOperationTimeout(int operationTimeout) {
    this.operationTimeout = operationTimeout;
}
public int getOperationTimeout(){
    return this.operationTimeout;
}


读后有收获可以支付宝请作者喝咖啡