`
sunwinner
  • 浏览: 197417 次
  • 性别: Icon_minigender_1
  • 来自: 上海
社区版块
存档分类
最新评论

Install hadoop-lzo-lib for Hadoop v1.1.1 on Mac OS X with Mountain Lion

阅读更多
Here is a quick guide to install hadoop lzo library, the install process is pretty easy actually.

 

  • Install LZO with HomeBrew, LZO is a wonderful compression scheme to use with Hadoop. HomeBrew is prerequisite, visit https://github.com/mxcl/homebrew to see how to install HomeBrew.
brew install lzo
  
  • Clone the hadoop-lzo source code to local:
git clone https://github.com/twitter/hadoop-lzo.git ~/code/hadoop-lzo && cd ~/code/hadoop-lzo
 
  • Build hadoop-lzo to generate hadoop-lzo-lib as well as hadoop lzo native library, find the built artifacts in the target directory.
C_INCLUDE_PATH=/usr/local/Cellar/lzo/2.06/include \
LIBRARY_PATH=/usr/local/Cellar/lzo/2.06/lib \
mvn clean test package
 Waiting for build complete. If something wrong, refer to https://github.com/twitter/hadoop-lzo for troubleshoting.
  • Copy the built artifacts into hadoop lib directory or native library directory respectively.

 

  • Once the libs are built and installed, you may want to add them to the class paths and library paths. That is, in hadoop-env.sh, set 
export HADOOP_CLASSPATH=$HADOOP_HOME/lib/hadoop-lzo-xxxx.jar
export JAVA_LIBRARY_PATH=/path/to/hadoop-lzo-native-libs:/path/to/standard-hadoop-native-libs

 

  • Note that there seems to be a bug in /path/to/hadoop/bin/hadoop; (refer to https://issues.apache.org/jira/browse/HADOOP-6453) . Comment out the line
    JAVA_LIBRARY_PATH=''

 

See https://github.com/twitter/hadoop-lzo for further instruction if you have any question or leave a comment.

分享到:
评论

相关推荐

    hadoop-lzo-0.4.20.jar

    cp /opt/hadoopgpl/lib/hadoop-lzo-0.4.20-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/ cp /opt/hadoopgpl/lib/hadoop-lzo-0.4.20-SNAPSHOT.jar $HBASE_HOME/lib/ 2.2.6. 拷贝本地库so文件到hadoop tar -cBf - ...

    hadoop-lzo-0.4.20-SNAPSHOT.jar

    编译后的hadoop-lzo源码,将hadoop-lzo-0.4.21-SNAPSHOT.jar放到hadoop的classpath下 如${HADOOP_HOME}/share/hadoop/common。hadoop才能正确支持lzo,免去编译的烦恼

    hadoop-lzo-0.4.21-SNAPSHOT jars

    Mac下编译的hadoop-lzo,Hadoop对应的版本是2.8.0,大家可以下载使用。

    hadoop-lzo-master

    将生成的 build/hadoop-lzo-0.4.15.jar cp 到 /usr/local/hadoop-1.0.2/lib 测试解压程序 bin/hadoop jar /usr/local/hadoop-1.0.2/lib/hadoop-lzo-0.4.15.jar ...

    hadoop-lzo-0.4.21-SNAPSHOT.jar

    hadoop-lzo-0.4.21-SNAPSHOT.jar是hadoop数据压缩lzo工具包

    hadoop-lzo-0.4.13.jar

    hadoop-lzo-0.4.13.jar 依赖包 hadoop-lzo-0.4.13.jar 依赖包 hadoop-lzo-0.4.13.jar 依赖包

    hadoop-lzo-0.4.15.jar

    hadoop2 lzo 文件 ,编译好的64位 hadoop-lzo-0.4.15.jar 文件 ,在mac 系统下编译的,用法:解压后把hadoop-lzo-0.4.15.jar 放到你的hadoop 安装路径下的lib 下,把里面lib/Mac_OS_X-x86_64-64 下的所有文件 拷到 ...

    hadoop-lzo-master.zip

    3. 安装LZO-2.x库(上步) 4. 编译hadoop-lzo $>C_INCLUDE_PATH=/usr/local/include $>LIBRARY_PATH=/usr/local/lib $>参见下文先搭建maven环境 $>mvn clean test(没有mvn则先走下步安装maven) $>出现lzo...

    hadoop-lzo-0.4.15.tar.gz

    hadoop-lzo-0.4.15.tar.gz

    hadoop-lzo-0.4.20-SNAPSHOT.jar 包

    hadoop lzo 压缩jar包,本人已经编译好,提供大家下载。

Global site tag (gtag.js) - Google Analytics