Hadoop本地库编译

  |   0 评论   |   1,222 浏览

一、基础库的编译

在${compile-home}如:/usr/local/src/hadoop-2.3.0-cdh5.0.0/src下执行编译语句:

mvn package -Pdist,native -DskipTests -Dtar -rf :hadoop-common -Drequire.snappy

编译完成后,二进制文件生成在

${compile-home}/hadoop-dist/target/hadoop-2.3.0-cdh5.0.0/lib/native下

列表如下:

libhadoop.a
libhadooppipes.a
libhadoop.so -> libhadoop.so.1.0.0
libhadoop.so.1.0.0
libhadooputils.a
libhdfs.a
libhdfs.so -> libhdfs.so.0.0.0
libhdfs.so.0.0.0

二、Hadoop Snappy编译安装方法:

下载源码

svn checkout https://github.com/google/snappy
cd /usr/local/hadoop-snappy-read-only
mvn package

err:/usr/bin/ld: cannot find -ljvm

解决办法:ln -s /usr/local/jdk/jre/lib/amd64/server/libjvm.so /usr/local/lib/

拷贝

cd /usr/local/hadoop-snappy-read-only/target/
tar zxvf hadoop-snappy-0.0.1-SNAPSHOT.tar.gz
cd hadoop-snappy-0.0.1-SNAPSHOT/lib
cp hadoop-snappy-0.0.1-SNAPSHOT.jar /usr/local/hadoop/lib
cp -r native/Linux-amd64-64/* /usr/local/hadoop/lib/native/

本地库列表

libhadoopsnappy.so -> libhadoopsnappy.so.0.0.1
libhadoopsnappy.so.0 -> libhadoopsnappy.so.0.0.1
libhadoopsnappy.so.0.0.1
libjvm.so -> /usr/local/jdk/jre/lib/amd64/server/libjvm.so
libprotobuf-lite.so -> libprotobuf-lite.so.8.0.0
libprotobuf-lite.so.8 -> libprotobuf-lite.so.8.0.0
libprotobuf-lite.so.8.0.0
libprotobuf.so -> libprotobuf.so.8.0.0
libprotobuf.so.8 -> libprotobuf.so.8.0.0
libprotobuf.so.8.0.0
libprotoc.so -> libprotoc.so.8.0.0
libprotoc.so.8 -> libprotoc.so.8.0.0
libprotoc.so.8.0.0
libsnappy.so -> libsnappy.so.1.2.0
libsnappy.so.1 -> libsnappy.so.1.2.0
libsnappy.so.1.2.0

Jar包

/usr/local/hadoop-snappy-read-only/target/hadoop-snappy-0.0.1-SNAPSHOT.jar

三、安装hadoop-lzo

Yum安装

cd /etc/yum.repos.d/
wget http://archive.cloudera.com/gplextras5/redhat/5/x86_64/gplextras/cloudera-gplextras5.repo
yum install hadoop-lzo

本地库列表

ll /usr/lib/hadoop/lib/native/
libgplcompression.a
libgplcompression.la
libgplcompression.lai
libgplcompression.so
libgplcompression.so.0
libgplcompression.so.0.0.0

Jar包

/usr/lib/hadoop/lib/hadoop-lzo-0.4.15-cdh5.1.3.jar

四、安装native-lzo

获得库文件(只需安装一次,得到库文件即可)

安装lzo

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz
./configure
make && make install

可得到:

/usr/lib64/liblzo2.so.2
/usr/lib64/liblzo2.so.2.0.0

安装gpl

wget https://hadoop-gpl-packing.googlecode.com/files/hadoop-gpl-packaging-0.6.1-1.x86_64.rpm
rpm -ivh hadoop-gpl-packaging-0.6.1-1.x86_64.rpm

可得到:

/opt/hadoopgpl

安装lzop(lzop为实现了lzo算法的压缩工具,不需要每台安装)

wget http://www.lzop.org/download/lzop-1.03.tar.gz
./configure
make && make install

可得到:

/usr/bin/lzop

拷贝(需拷贝至每个节点)

cp /usr/lib64/liblzo2.so.2 /usr/lib64/liblzo2.so.2.0.0 /usr/local/hadoop/lib/native/
cp /opt/hadoopgpl/native/Linux-amd64-64/* /usr/local/hadoop/lib/native/
cp /opt/hadoopgpl/lib/hadoop-lzo-0.4.17.jar /opt/hadoopgpl/lib/yamlbeans-0.9.3.jar /usr/local/hadoop/lib

本地库列表:

libgplcompression.a
libgplcompression.la
libgplcompression.so
libgplcompression.so.0
libgplcompression.so.0.0.0
LzoCompressor.lo
LzoCompressor.o
LzoDecompressor.lo
LzoDecompressor.o

Jar包:

/opt/hadoopgpl/lib/hadoop-lzo-0.4.17.jar
/opt/hadoopgpl/lib/yamlbeans-0.9.3.jar


读后有收获可以支付宝请作者喝咖啡