搜索
创作
提问
登录注册
展开

专栏首页魏晓蕾的专栏Apache 及 CDH Spark 源码编译

Apache 及 CDH Spark 源码编译

阅读 4540

1、Apache Spark 源码编译

软件版本： JDK：1.7.0_67 Scala：2.10.4 Hadoop：2.5.0 Spark：1.6.1 Maven：3.3.3 Zinc：0.3.5.3

（1）搭建Maven环境 1）解压Maven安装包 maven安装包下载地址：http://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/ cd /opt/softres/ softwares]$ tar -zxf apache-maven-3.3.3-bin.tar.gz -C /opt/modules/ 2）修改配置文件 apache-maven-3.3.3]$ cd conf conf]$ vim settings.xml

<mirrors>
    <mirror>
      <id>aliyun</id>
      <mirrorOf>central</mirrorOf>
      <name>aliyun repository</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
  </mirrors>

3）配置MAVEN_HOME环境变量 $ vim /etc/profile

# MAVEN_HOME
export MAVEN_HOME=/opt/modules/apache-maven-3.3.3
export PATH=$PATH:$MAVEN_HOME/bin

（2）搭建Spark环境 1）解压spark源码包 spark源码包下载地址：http://archive.apache.org/dist/spark/spark-1.6.1/ softwares]$ tar -zxf spark-1.6.1.tgz -C /opt/modules/ 2）修改配置文件/opt/modules/spark-1.6.1/make-distribution.sh

# Figure out where the Spark framework is installed
SPARK_HOME=/opt/modules/spark-1.6.1
DISTDIR="$SPARK_HOME/dist"

VERSION=1.6.1
SCALA_VERSION=2.10.4
SPARK_HADOOP_VERSION=2.5.0
SPARK_HIVE=1

3）修改配置文件/opt/modules/spark-1.6.1/pom.xml中的hadoop.version和scala.version

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <akka.group>com.typesafe.akka</akka.group>
    <akka.version>2.3.11</akka.version>
    <java.version>1.7</java.version>
    <maven.version>3.3.3</maven.version>
    <sbt.project.name>spark</sbt.project.name>
    <mesos.version>0.21.1</mesos.version>
    <mesos.classifier>shaded-protobuf</mesos.classifier>
    <slf4j.version>1.7.10</slf4j.version>
    <log4j.version>1.2.17</log4j.version>
    <hadoop.version>2.5.0</hadoop.version>
    <protobuf.version>2.5.0</protobuf.version>
    <yarn.version>${hadoop.version}</yarn.version>
    <hbase.version>0.98.7-hadoop2</hbase.version>
    <hbase.artifact>hbase</hbase.artifact>
    <flume.version>1.6.0</flume.version>
    <zookeeper.version>3.4.5</zookeeper.version>
    <curator.version>2.4.0</curator.version>
    <hive.group>org.spark-project.hive</hive.group>
    <!-- Version used in Maven Hive dependency -->
        <hive.version>1.2.1.spark</hive.version>
    <!-- Version used for internal directory structure -->
    <hive.version.short>1.2.1</hive.version.short>
    <derby.version>10.10.1.1</derby.version>
    <parquet.version>1.7.0</parquet.version>
    <hive.parquet.version>1.6.0</hive.parquet.version>
    <jblas.version>1.2.4</jblas.version>
    <jetty.version>8.1.14.v20131031</jetty.version>
    <orbit.version>3.0.0.v201112011016</orbit.version>
    <chill.version>0.5.0</chill.version>
    <ivy.version>2.4.0</ivy.version>
    <oro.version>2.0.8</oro.version>
    <codahale.metrics.version>3.1.2</codahale.metrics.version>
    <avro.version>1.7.7</avro.version>
    <avro.mapred.classifier>hadoop2</avro.mapred.classifier>
    <jets3t.version>0.7.1</jets3t.version>
    <aws.kinesis.client.version>1.4.0</aws.kinesis.client.version>
    <!-- the producer is used in tests -->
    <aws.kinesis.producer.version>0.10.1</aws.kinesis.producer.version>
    <!--  org.apache.httpcomponents/httpclient-->
    <commons.httpclient.version>4.3.2</commons.httpclient.version>
        <!--  commons-httpclient/commons-httpclient-->
    <httpclient.classic.version>3.1</httpclient.classic.version>
    <commons.math3.version>3.4.1</commons.math3.version>
    <!-- managed up from 3.2.1 for SPARK-11652 -->
    <commons.collections.version>3.2.2</commons.collections.version>
    <scala.version>2.10.4</scala.version>
    <scala.binary.version>2.10</scala.binary.version>
    <jline.version>${scala.version}</jline.version>
    <jline.groupid>org.scala-lang</jline.groupid>
    <codehaus.jackson.version>1.9.13</codehaus.jackson.version>
    <fasterxml.jackson.version>2.4.4</fasterxml.jackson.version>
    <snappy.version>1.1.2</snappy.version>
    <netlib.java.version>1.1.2</netlib.java.version>
    <calcite.version>1.2.0-incubating</calcite.version>
    <commons-codec.version>1.10</commons-codec.version>
    <!-- org.apache.commons/commons-lang/-->
    <commons-lang2.version>2.6</commons-lang2.version>
    <!-- org.apache.commons/commons-lang3/-->
    <commons-lang3.version>3.3.2</commons-lang3.version>
    <datanucleus-core.version>3.2.10</datanucleus-core.version>
    <janino.version>2.7.8</janino.version>
    <jersey.version>1.9</jersey.version>
    <joda.version>2.9</joda.version>
        <jodd.version>3.5.2</jodd.version>
    <jsr305.version>1.3.9</jsr305.version>
    <libthrift.version>0.9.2</libthrift.version>

    <test.java.home>${java.home}</test.java.home>
    <test.exclude.tags></test.exclude.tags>

4）将编译用到的scala-2.10.4.zip和zinc-0.3.5.3.tgz文件解压到/opt/modules/spark-1.6.1/build目录下（zinc-0.3.5.3.tgz文件下载地址：http://downloads.typesafe.com/zinc/0.3.5.3/zinc-0.3.5.3.tgz）。 build]$ unzip /opt/softwares/scala-2.10.4.zip build]$ tar -zxf /opt/softwares/zinc-0.3.5.3.tgz （3）编译Spark spark-1.6.1]$ ./make-distribution.sh --tgz -Phadoop-2.4 -Dhadoop.version=2.5.0 -Pyarn -Phive -Phive-thriftserver （4）编译过程及编译成功截图如下

Apache Hadoop-2.5.0-Spark-1.6.1编译成功。

2、Apache Spark 源码编译过程中报错总结

【报错1】

Using `mvn` from path: /opt/modules/apache-maven-3.3.3/bin/mvn
[ERROR] Error executing Maven.
[ERROR] 1 problem was encountered while building the effective settings
[FATAL] Non-parseable settings /opt/modules/apache-maven-3.3.3/conf/settings.xml: Duplicated tag: 'mirrors' (position: START_TAG seen ...</mirrors>\n\n  <mirrors>... @161:12)  @ /opt/mules/apache-maven-3.3.3/conf/settings.xml, line 161, column 12

【解决方案】报错标签重复，将 /opt/modules/apache-maven-3.3.3/conf/settings.xml文件中的已有部分去掉，保留我配置的aliyun镜像即可。 原settings.xml文件：

<mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
    <mirror>
      <id>mirrorId</id>
      <mirrorOf>repositoryId</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://my.repository.com/repo/path</url>
    </mirror>
     -->
  </mirrors>

  <mirrors>
    <mirror>
      <id>aliyun</id>
      <mirrorOf>central</mirrorOf>
      <name>aliyun repository</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
  </mirrors>

改后的settings.xml文件：

  <mirrors>
    <mirror>
      <id>aliyun</id>
      <mirrorOf>central</mirrorOf>
      <name>aliyun repository</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
  </mirrors>

3、CDH Spark 源码编译

软件版本： JDK：1.7.0_67 Scala：2.10.4 Hadoop：2.5.0-cdh5.3.6 Spark：1.6.1 Maven：3.3.3 Zinc：0.3.5.3

（1）备份MAVEN环境 cd /home/beifeng ~]$ mkdir m2-apache-apark-backup ~]$ cp -r ./.m2/* m2-apache-apark-backup/ cd /home/beifeng/.m2 .m2]$ rm -rf ./* （2）搭建Spark环境 1）解压spark源码包 softwares]$ tar -zxf spark-1.6.1.tgz softwares]$ mv spark-1.6.1 /opt/modules/spark-1.6.1-cdh5.3.6 2）修改配置文件/opt/modules/spark-1.6.1/pom.xml中的组件为cdh-5.3.6版本

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <akka.group>com.typesafe.akka</akka.group>
    <akka.version>2.3.11</akka.version>
    <java.version>1.7</java.version>
    <maven.version>3.3.3</maven.version>
    <sbt.project.name>spark</sbt.project.name>
    <mesos.version>0.21.1</mesos.version>
    <mesos.classifier>shaded-protobuf</mesos.classifier>
    <slf4j.version>1.7.10</slf4j.version>
    <log4j.version>1.2.17</log4j.version>
    <hadoop.version>2.5.0-cdh5.3.6</hadoop.version>
    <protobuf.version>2.5.0</protobuf.version>
    <yarn.version>${hadoop.version}</yarn.version>
        <hbase.version>0.98.7-hadoop2</hbase.version>
    <hbase.artifact>hbase</hbase.artifact>
    <flume.version>1.6.0-cdh5.3.6</flume.version>
    <zookeeper.version>3.4.5-cdh5.3.6</zookeeper.version>
    <curator.version>2.4.0</curator.version>
    <hive.group>org.spark-project.hive</hive.group>
    <!-- Version used in Maven Hive dependency -->
    <hive.version>1.2.1.spark</hive.version>
    <!-- Version used for internal directory structure -->
    <hive.version.short>1.2.1</hive.version.short>
    <derby.version>10.10.1.1</derby.version>
    <parquet.version>1.7.0</parquet.version>
    <hive.parquet.version>1.6.0</hive.parquet.version>
    <jblas.version>1.2.4</jblas.version>
    <jetty.version>8.1.14.v20131031</jetty.version>
    <orbit.version>3.0.0.v201112011016</orbit.version>
    <chill.version>0.5.0</chill.version>
    <ivy.version>2.4.0</ivy.version>
    <oro.version>2.0.8</oro.version>
    <codahale.metrics.version>3.1.2</codahale.metrics.version>
    <avro.version>1.7.7</avro.version>
    <avro.mapred.classifier>hadoop2</avro.mapred.classifier>
    <jets3t.version>0.7.1</jets3t.version>
    <aws.kinesis.client.version>1.4.0</aws.kinesis.client.version>
        <!-- the producer is used in tests -->
    <aws.kinesis.producer.version>0.10.1</aws.kinesis.producer.version>
    <!--  org.apache.httpcomponents/httpclient-->
    <commons.httpclient.version>4.3.2</commons.httpclient.version>
    <!--  commons-httpclient/commons-httpclient-->
    <httpclient.classic.version>3.1</httpclient.classic.version>
    <commons.math3.version>3.4.1</commons.math3.version>
    <!-- managed up from 3.2.1 for SPARK-11652 -->
    <commons.collections.version>3.2.2</commons.collections.version>
    <scala.version>2.10.4</scala.version>
    <scala.binary.version>2.10</scala.binary.version>
    <jline.version>${scala.version}</jline.version>
    <jline.groupid>org.scala-lang</jline.groupid>
    <codehaus.jackson.version>1.9.13</codehaus.jackson.version>
    <fasterxml.jackson.version>2.4.4</fasterxml.jackson.version>
    <snappy.version>1.1.2</snappy.version>
    <netlib.java.version>1.1.2</netlib.java.version>
    <calcite.version>1.2.0-incubating</calcite.version>
    <commons-codec.version>1.10</commons-codec.version>
    <!-- org.apache.commons/commons-lang/-->
    <commons-lang2.version>2.6</commons-lang2.version>
    <!-- org.apache.commons/commons-lang3/-->
    <commons-lang3.version>3.3.2</commons-lang3.version>
        <datanucleus-core.version>3.2.10</datanucleus-core.version>
    <janino.version>2.7.8</janino.version>
    <jersey.version>1.9</jersey.version>
    <joda.version>2.9</joda.version>
    <jodd.version>3.5.2</jodd.version>
    <jsr305.version>1.3.9</jsr305.version>
    <libthrift.version>0.9.2</libthrift.version>

    <test.java.home>${java.home}</test.java.home>
    <test.exclude.tags></test.exclude.tags>

3）修改配置文件/opt/modules/spark-1.6.1/make-distribution.sh

VERSION=1.6.1
SCALA_VERSION=2.10.4
SPARK_HADOOP_VERSION=2.5.0-cdh5.3.6
SPARK_HIVE=1

4）修改配置文件/opt/modules/apache-maven-3.3.3/conf/settings.xml

  <mirrors>
    <mirror>
      <id>cloudera-repo</id>
      <mirrorOf>central</mirrorOf>
      <name>Cloudera Repository</name>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
      <releases>
          <enabled>true</enabled>
      </releases>
      <snapshots>
          <enabled>false</enabled>
      </snapshots>
    </mirror>
    <mirror>
      <id>aliyun</id>
      <mirrorOf>central</mirrorOf>
      <name>aliyun repository</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
  </mirrors>

5）将scala和zinc的安装包解压到/opt/modules/spark-1.6.1-cdh5.3.6/build/目录下 build]$ unzip /opt/softwares/scala-2.10.4.zip -d /opt/modules/spark-1.6.1-cdh5.3.6/build/ build]$ tar -zxf /opt/softwares/zinc-0.3.5.3.tgz -C . （3）编译Spark spark-1.6.1-cdh5.3.6]$ ./make-distribution.sh --tgz -Phadoop-2.4 -Dhadoop.version=2.5.0-cdh5.3.6 -Pyarn -Phive -Phive-thriftserver （4）编译失败，报错未能解决

4、CDH Spark 源码编译过程中报错总结

【报错1】

Using `mvn` from path: /opt/modules/apache-maven-3.3.3/bin/mvn
[ERROR] Error executing Maven.
[ERROR] 1 problem was encountered while building the effective settings
[FATAL] Non-parseable settings /opt/modules/apache-maven-3.3.3/conf/settings.xml: end tag name </mirrors> must match start tag name <mirror> from line 154 (position: TEXT seen ...</snapshots>\n  </mirrors>... @164:13)  @ /opt/modules/apache-maven-3.3.3/conf/settings.xml, line 164, column 13

【解决方案】

    <mirror>
      <id>cloudera-repo</id>
      <name>Cloudera Repository</name>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
      <releases>
          <enabled>true</enabled>
      </releases>
      <snapshots>
          <enabled>false</enabled>
      </snapshots>
    </mirror>

【报错2】

Using `mvn` from path: /opt/modules/apache-maven-3.3.3/bin/mvn
[ERROR] Error executing Maven.
[ERROR] 2 problems were encountered while building the effective settings
[WARNING] Unrecognised tag: 'releases' (position: START_TAG seen ...</url>\n      <releases>... @158:17)  @ /opt/modules/apache-maven-3.3.3/conf/settings.xml, line 158, column 17
[ERROR] 'mirrors.mirror.mirrorOf' for cloudera-repo is missing @ /opt/modules/apache-maven-3.3.3/conf/settings.xml

【解决方案】

  <mirrors>
    <mirror>
      <id>aliyun</id>
      <mirrorOf>central</mirrorOf>
      <name>aliyun repository</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
    <mirror>
      <id>cloudera-repo</id>
      <mirrorOf>central</mirrorOf>
      <name>Cloudera Repository</name>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
      <releases>
          <enabled>true</enabled>
      </releases>
      <snapshots>
          <enabled>false</enabled>
      </snapshots>
    </mirror>
  </mirrors>

【报错3】

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-launcher_2.10: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.10:jar:1.6.1: Could not find artifact org.apache.hadoop:hadoop-client:jar:2.5.0-cdh5.3.6 in aliyun (http://maven.aliyun.com/nexus/content/groups/public/) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-launcher_2.10

【解决方案】

  <mirrors>
    <mirror>
      <id>cloudera-repo</id>
      <mirrorOf>central</mirrorOf>
      <name>Cloudera Repository</name>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
      <releases>
          <enabled>true</enabled>
      </releases>
      <snapshots>
          <enabled>false</enabled>
      </snapshots>
    </mirror>
    <mirror>
      <id>aliyun</id>
      <mirrorOf>central</mirrorOf>
      <name>aliyun repository</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
  </mirrors>

【报错4】

[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-remote-resources-plugin:jar:1.5: Could not transfer artifact org.apache.maven.plugins:maven-remote-resources-plugin:pom:1.5 from/to cloudera-repo (https://repository.cloudera.com/artifactory/cloudera-repos): Remote host closed connection during handshake: SSL peer shut down incorrectly -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

【解决方案】

  <mirrors>
    <mirror>
      <id>cloudera-repo</id>
      <mirrorOf>central</mirrorOf>
      <name>Cloudera Repository</name>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
    </mirror>
    <mirror>
      <id>aliyun</id>
      <mirrorOf>*</mirrorOf>
      <name>aliyun repository</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
  </mirrors>

本文参与腾讯云自媒体分享计划，欢迎正在阅读的你也加入，一起分享。

展开阅读全文
举报

我来说两句

0 条评论
登录 后参与评论

相关文章

  • JeecgBoot配置及前后端分离HelloWorld项目

    JEECG存在自定义JAR包，放在自己的Maven私服上面，所以有的时候会遇到下载失败。一般遇到下载失败的情况，是因为用户设置了本地镜像，导致无法从JEECG私...

    魏晓蕾

  • 【Hadoop】Linux环境下运行Eclipse并配置搭建Maven环境详细步骤

    版权声明：本文为博主原创文章，转载请注明出处。 https://blog.csdn.net/...

    魏晓蕾

  • 【NDN IoT】

    物联网正处于婴儿期，与万维网使因特网成为一种必不可少的通讯、商业和娱乐工具之前因特网的状态相似。为使得物联网变得实际有用，我们需要一种相似的革新。这...

    魏晓蕾

  • （七）整合spring cloud云服务架构 - common-service 项目构建过程

    我们将对common-service整个项目进行剖析，将整个构建的流程给记录下来，让更多的关注者来参考学习。

    用户7788846

  • spring boot 中的 Parent POM 和 Starter 的作用什么？（☆☆☆）

    spring boot 推荐在 maven 中，继承 spring-boot-starter-parent 来作为父项目。因为继承 spring-boot-st...

    水货程序员

  • org.hibernate.type.StringType cannot be cast to org.hibernate.type.VersionType

    两种情况，一种是有关键字不能解析 某张表有version字段，从hibernate tools生成的hbm文件内容： <version name="versio...

    cloudskyme

  • spring boot整合dubbox进行服务拆分

    Dubbo是一个来自阿里巴巴的开源分布式服务框架，当当根据自身的需求，为Dubbo实现了一些新的功能，包括REST风格远程调用、Kryo/FST序列化等等。并...

    猿天地

  • Maven WEB 项目使用ProGuard进行混淆，最佳解决方案

    近期公司的Android项目做了混淆，虽说对于保护代码并不是100%的，但混淆后的代码可以使那些不法份子难以阅读，这样也能对代码的保护做出贡献。  于是，公司写...

    庞小明

  • Java 实现微信宝等多个支付平台合一的二维码支付( ssm 框架)

    实现微信支付宝等支付平台合多为一的二维码支付，并且实现有效时间内支付有效，本人采用的框架是spring springmvc mybatis 框架，maven管理...

    好好学java

  • 支持CDS view的SAP ALV IDA - 使用前提

    CHECK cl_salv_gui_table_ida=>db_capabilities( )->is_table_supported( iv_ddic_tab...

    Jerry Wang
更多文章

作者介绍

魏晓蕾

专栏

精选专题

活动推荐

运营活动

活动名称
广告关闭

扫码关注云+社区

领取腾讯云代金券