spark编译:构建基于hadoop的spark安装包及遇到问题总结

问题导读 1.spark集群能否单独运行? 2.如何在spark中指定想编译的hadoop版本? 3.构建的时候,版本选择需要注意什么? 上一篇 如何查看spark与hadoop、kafka、Scala、flume、hive等兼容版本【适用于任何版本】 http://www.aboutyun.com/forum.php?mod=viewthread&tid=23252 构建基于hadoop的spark安装包

上一篇说了spark与其它组件兼容的版本,这里具体说下如何构建基于hadoop的spark安装包。首先我们需要有spark源码,上一篇已经交给大家如何使用git下载。当然我这里提供了百度网盘链接 链接:http://pan.baidu.com/s/1gfMpTqb 密码:c6dc 默认情况下,spark的执行不需要hadoop,也就是说没有hadoop集群下,spark集群可以设置,启动,使用。。尽管如此,如果你使用spark如Yarn的执行模式,或则访问hdfs创建rdd,它将会依赖hadoop。如果是这种情况,你的spark安装包必须兼容你所使用的hadoop集群的安装包 如果你使用的是spark2.3.0对应的hadoop默认为2.6.在假如使用的是spark1.2.0对应的是hadoop2.4. 这里需要注意的是:有些hadoop版本,是有小版本的,比如hadoop2.6.5,hadoop2.7有hadoop2.7.1,hadoop2.7.3.对于hadoop版本的+或则-的小版本之间,它们与spark大多都是兼容可以正常运行的。 spark构建运行开发者指定hadoop版本,你可以直接使用maven命令或则 make-distribution.sh脚本生成安装包。例如构建spark2.3.0,hadoop2.6 maven profile 使用-P选项, 如: $SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6 -Phive spark2.3.0支持2.6,2.7 如何自定义hadoop版本 基本的你需要在pom文件中添加profile指定hadoop版本 假如你想构建 Hadoop 2.6.5,按照下面步骤 第一步: 在 $SPARK_SRC/pom.xml中添加maven profile hadoop-2.6.5 在<profiles> 部分

[XML] 纯文本查看 复制代码

?

<profile>
    <id>hadoop-2.6.5</id>
    <properties>
        <hadoop.version>2.6.5</hadoop.version>
        <protobuf.version>2.5.0</protobuf.version>
        <jets3t.version>0.9.0</jets3t.version>
        <commons.math3.version>3.1.1</commons.math3.version>
        <avro.mapred.classifier>hadoop2</avro.mapred.classifier>
    </properties>
</profile>

protobuf, jets3t, commons.math3 和avro.mapred.classifier 版本都是来自Apache Hadoop 2.5.2 二进制分布式libs. 你需要修改要构建的hadoop的对应版本。

对于这个avro.mapred.classifier,大家可以找找,不过在spark2.3.0 pom文件中也是有的 https://github.com/apache/spark/blob/master/pom.xml

上面是比较复杂的,我们也可以使用最简单的方式:

[XML] 纯文本查看 复制代码

?

<profile>
    <id>hadoop-2.6.5</id>
</profile>

第二步:构建执行 使用make-distribution.sh构建

[Bash shell] 纯文本查看 复制代码

?

$SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6.5 -Dhadoop.version=2.6.5 -Phive

使用maven命令构建

[Bash shell] 纯文本查看 复制代码

?

mvn -Pyarn -Dhadoop.version=2.6.5 -Dyarn.version=2.6.5 -DskipTests clean package

构建成功后,可以找到spark-2.*.*-bin-2.6.5.tgz 注意: 这种方式建议使用在hadoop小版本,对于hadoop主版本即使构建成功,也可能在生产中遇到一些问题。 参考: https://etushar89.wordpress.com/2014/12/28/building-apache-spark-against-specific-hadoop-version/

构建基于hadoop的spark安装包实践及遇到问题总结

spark编译时间还是比较长的,可能要一两个小时,而且有时候卡住那不动。 在编译的过程中,有编译失败,也有编译成功的。总之两个条件: 1.跟版本有关系 版本不对,可能编译失败 2.跟网速有关系 网速不好,有些下载失败,导致编译失败。 使用hadoop2.6.5

[Bash shell] 纯文本查看 复制代码

?

$SPARK_SRC/make-distribution.sh --tgz -Pyarn -Phadoop-2.6.5 -Dhadoop.version=2.6.5 -Phive

这里使用的是spark2.3.0,hadoop版本为2.6.5,最后编译失败。报错如下

[Bash shell] 纯文本查看 复制代码

?

[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Skipping Spark Integration for Kafka 0.10 Assembly
[INFO] This project has been banned from the build due to previous failures.
[INFO] ------------------------------------------------------------------------
[INFO] Spark Project Parent POM ........................... SUCCESS [08:36 min]
[INFO] Spark Project Tags ................................. SUCCESS [04:56 min]
[INFO] Spark Project Sketch ............................... SUCCESS [ 10.413 s]
[INFO] Spark Project Local DB ............................. SUCCESS [01:15 min]
[INFO] Spark Project Networking ........................... SUCCESS [ 36.812 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 11.964 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 34.261 s]
[INFO] Spark Project Launcher ............................. SUCCESS [02:40 min]
[INFO] Spark Project Core ................................. SUCCESS [12:34 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [02:19 min]
[INFO] Spark Project GraphX ............................... SUCCESS [01:29 min]
[INFO] Spark Project Streaming ............................ SUCCESS [01:52 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [05:25 min]
[INFO] Spark Project SQL .................................. FAILURE [10:13 min]
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SUCCESS [ 17.955 s]
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 19.981 s]
[INFO] Spark Project YARN ................................. SUCCESS [01:14 min]
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:33 min]
[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 56:26 min
[INFO] Finished at: 2017-11-08T10:44:58+08:00
[INFO] Final Memory: 65M/296M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "hadoop-2.6.5" could not be activated because it does not exist.
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed. CompileFailed -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-sql_2.11

有一个错误和警告是比较关键的 [WARNING] The requested profile "hadoop-2.6.5" could not be activated because it does not exist. [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: 上面是profile中是没有hadoop-2.6.5,因此我们需要增加profile

[XML] 纯文本查看 复制代码

?

<profile>
    <id>hadoop-2.6.5</id>
</profile>

第二个问题不能执行,这个可能就跟网速有关系。多次执行仍然失败,也可能跟版本有关系 更换hadoop2.7.1 上面失败,接着我们尝试hadoop2.7.1

[XML] 纯文本查看 复制代码

?

<profile>
    <id>hadoop-2.6.5</id>
</profile>

编译成功 但是报错如下

[Bash shell] 纯文本查看 复制代码

?

+ TARDIR_NAME='spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ TARDIR='/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ rm -rf '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ cp -r /home/aboutyun/spark/dist '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ tar czf 'spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist..tgz' -C /home/aboutyun/spark 'spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'
+ rm -rf '/home/aboutyun/spark/spark-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.-bin-[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.'

[WARNING] The requested profile "hadoop-2.7.1" could not be activated because it does not exist.这句很关键,意思是虽然编译成功,但是不能使用。在后面将截图给大家看。 所以在pom.xml文件中添加如下属性

[Bash shell] 纯文本查看 复制代码

?

[mw_shl_code=bash,true]<profile>
<id>hadoop-2.7.1</id>
</profile>

[/mw_shl_code] 编译成功

下面我们通过winscp查看第一个为未添加profile,第二个添加后,编译成功。

原文发布于微信公众号 - about云(wwwaboutyuncom)

原文发表时间:2017-11-09

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏一个会写诗的程序员的博客

第11章 Spring Boot应用监控第11章 Spring Boot应用监控小结

在实际的生产系统中,我们怎样知道我们的应用运行良好呢?我们往往需要对系统实际运行的情况(各种cpu,io,disk,db,业务功能等指标)进行监控运维。这需要耗...

3063
来自专栏Hadoop实操

如何使用Sentry为Kafka赋权

8454
来自专栏bboysoul

linux 格式化硬盘

fdisk /dev/sda 输入o建立dos分区表 输入n建立一个新的分区 输入p建立一个主分区,或者输入e建立一个逻辑分区 之后输入分区号码mbr只...

2275
来自专栏个人分享

spark单机模式简单搭建

待安装列表 hadoop hive scala spark 一.环境变量配置: ~/.bash_profile PATH=$PATH:$HOME/bin

5481
来自专栏Hadoop实操

非Kerberos环境下Kafka数据到Flume进Hive表

前面Fayson讲过《如何在Kerberos环境使用Flume采集Kafka数据并写入HDFS》,本篇文章主要讲述如何在非Kerberos环境下将Kafka数据...

6654
来自专栏安恒网络空间安全讲武堂

HackTheBox - Poison Writeup

来源:https://www.absolomb.com/2018-09-08-HackTheBox-Poison/

1412
来自专栏开发与安全

linux系统编程之管道(二):管道读写规则和Pipe Capacity、PIPE_BUF

一、当没有数据可读时 O_NONBLOCK disable:read调用阻塞,即进程暂停执行,一直等到有数据来到为止。 O_NONBLOCK enable:r...

3359
来自专栏用户2442861的专栏

ubuntu16安装nginx

https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-...

1742
来自专栏流柯技术学院

linux下安装rzsz

wget http://freeware.sgi.com/source/rzsz/rzsz-3.48.tar.gz

5771
来自专栏闵开慧

hive中配置hwi

vi $HIVE_HOME/conf/hive-site.xml 配置如下 <property>   <name>hive.hwi.listen.host<...

3546

扫码关注云+社区

领取腾讯云代金券