Azkaban-3.x源码编译以及Solo-Server模式部署

软件准备

环境准备

  • JDK-1.8+
  • Git:yum install git -y

1. 源码编译

(1) 解压源码包

# 解压后的目录结构如下:
[hadoop@beh07 azkaban-3.73.1]$ ll
total 92
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-core
drwxrwxr-x 4 hadoop hadoop    52 May 31 06:26 az-crypto
drwxrwxr-x 5 hadoop hadoop    86 May 31 06:26 az-examples
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-exec-util
drwxrwxr-x 3 hadoop hadoop    54 May 31 06:26 az-flow-trigger-dependency-plugin
drwxrwxr-x 3 hadoop hadoop    53 May 31 06:26 az-flow-trigger-dependency-type
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-hadoop-jobtype-plugin
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-hdfs-viewer
-rw-rw-r-- 1 hadoop hadoop 21925 May 31 06:26 az-intellij-style.xml
drwxrwxr-x 4 hadoop hadoop    49 May 31 06:26 az-jobsummary
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-common
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-db
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-exec-server
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 azkaban-hadoop-security-plugin
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-solo-server
drwxrwxr-x 3 hadoop hadoop    54 May 31 06:26 azkaban-spi
drwxrwxr-x 3 hadoop hadoop   100 May 31 06:26 azkaban-web-server
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-reportal
-rw-rw-r-- 1 hadoop hadoop 10672 May 31 06:26 build.gradle
-rw-rw-r-- 1 hadoop hadoop  6409 May 31 06:26 CONTRIBUTING.md
drwxrwxr-x 3 hadoop hadoop  4096 May 31 06:26 docs
drwxrwxr-x 3 hadoop hadoop    21 May 31 06:26 gradle
-rw-rw-r-- 1 hadoop hadoop  1488 May 31 06:26 gradle.properties
-rwxrwxr-x 1 hadoop hadoop  5296 May 31 06:26 gradlew
-rw-rw-r-- 1 hadoop hadoop  2260 May 31 06:26 gradlew.bat
-rw-rw-r-- 1 hadoop hadoop 11358 May 31 06:26 LICENSE
-rw-rw-r-- 1 hadoop hadoop  2359 May 31 06:26 NOTICE
-rw-rw-r-- 1 hadoop hadoop  2406 May 31 06:26 README.md
-rw-rw-r-- 1 hadoop hadoop    31 May 31 06:26 requirements.txt
-rw-rw-r-- 1 hadoop hadoop  1170 May 31 06:26 settings.gradle
drwxrwxr-x 6 hadoop hadoop   124 May 31 06:26 test
drwxrwxr-x 2 hadoop hadoop    78 May 31 06:26 tools

(2) 优化gradle

查看azkaban-3.73.1/gradle/wrapper/gradle-wrapper.properties文件:

distributionUrl=https\://services.gradle.org/distributions/gradle-4.6-all.zip

依赖的gradle为4.6版本,从https://gradle.org/releases/下载对应版本的gradle的zip包,放到azkaban-3.73.1/gradle/wrapper目录下。

然后修改gradle-wrapper.properties文件的最后一行,改为:

#distributionUrl=https\://services.gradle.org/distributions/gradle-4.6-all.zip
distributionUrl=gradle-4.6-all.zip

(3) 编译源码

在解压目录下有一个可执行文件gradlew,进行以下操作:

  • step 1,这一步是执行时间最长的,耐心等待
[hadoop@beh07 azkaban-3.73.1]$ ./gradlew build
Downloading https://services.gradle.org/distributions/gradle-4.6-all.zip
............

# 可能遇到的报错1:
Could not determine the dependencies of task ':az-flow-trigger-dependency-type:kafka-event-trigger:fatJar'.
> Could not resolve all files for configuration ':az-flow-trigger-dependency-type:kafka-event-trigger:compile'.
   > Could not download avro-tools.jar (org.apache.avro:avro-tools:1.8.1)
      > Could not get resource 'https://repo.maven.apache.org/maven2/org/apache/avro/avro-tools/1.8.1/avro-tools-1.8.1.jar'.
         > Read timed out
   > Could not download netty.jar (io.netty:netty:3.10.5.Final)
      > Could not get resource 'https://repo.maven.apache.org/maven2/io/netty/netty/3.10.5.Final/netty-3.10.5.Final.jar'.
         > Connection reset
# 解决办法:
重新执行./gradlew build,直到所需的jar包下载完成

# 可能遇到的报错2:
> Could not resolve net.jpountz.lz4:lz4:1.2.0.
  Required by:
      project :az-hadoop-jobtype-plugin > org.apache.spark:spark-core_2.10:1.4.0
   > Skipped due to earlier error
# 解决办法:
重新执行./gradlew build,直到所需的jar包下载完成

# 执行成功的消息:
BUILD SUCCESSFUL in 12m 24s
105 actionable tasks: 44 executed, 61 up-to-date
  • step 2
[hadoop@beh07 azkaban-3.73.1]$ ./gradlew clean
......
BUILD SUCCESSFUL in 2s
19 actionable tasks: 19 executed
  • step 3
[hadoop@beh07 azkaban-3.73.1]$ ./gradlew installDist
......
BUILD SUCCESSFUL in 11s
53 actionable tasks: 38 executed, 14 from cache, 1 up-to-date
  • step 4
[hadoop@beh07 azkaban-3.73.1]$ ./gradlew test
......
BUILD SUCCESSFUL in 1m 32s
68 actionable tasks: 13 executed, 5 from cache, 50 up-to-date

# 如果遇到报错的情况,尝试多执行几次
  • step 5
[hadoop@beh07 azkaban-3.73.1]$ ./gradlew build -x test
......
BUILD SUCCESSFUL in 21s
74 actionable tasks: 31 executed, 1 from cache, 42 up-to-date

执行完以上5步之后,我们就编译好了源码,需要的安装包也已经打包好了,去以下目录中找:

azkaban-3.73.1/azkaban-db/build/distributions
azkaban-3.73.1/azkaban-exec-server/build/distributions
azkaban-3.73.1/azkaban-hadoop-security-plugin/build/distributions
azkaban-3.73.1/azkaban-solo-server/build/distributions
azkaban-3.73.1/azkaban-web-server/build/distributions

找到的.tar.gz文件分别是:

azkaban-db-0.1.0-SNAPSHOT.tar.gz
azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.tar.gz
azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
azkaban-web-server-0.1.0-SNAPSHOT.tar.gz

这样,一份新鲜的azkaban安装包就已经出炉了!

2. Solo-Server模式安装

在Azkaban-3.0之后,有两种部署模式:

Solo-Server模式:使用内置的数据库H2,并且Web服务器和Executor服务器都在同一进程中运行。

Multiple-Executor模式:这种模式适用于正式的生产环境。数据库应该由具有主从设置的MySQL实例支持。理想情况下,Web服务器和Executor服务器应在不同的主机中运行,这样升级和维护的时候就不会影响用户。这种多主机设置为Azkaban带来了强大且可扩展的特性。

solo-server模式部署过程:

  • 解压azkaban-solo-server安装包,解压后的目录结构如下:
[hadoop@beh07 azkaban-solo-server]$ ll
total 8
drwxr-xr-x 3 hadoop hadoop   67 Aug 21 14:40 bin
drwxr-xr-x 2 hadoop hadoop   82 Aug 21 14:40 conf
drwxr-xr-x 2 hadoop hadoop 4096 Aug 21 14:40 lib
drwxr-xr-x 3 hadoop hadoop   22 Aug 21 14:40 plugins
drwxr-xr-x 2 hadoop hadoop 4096 Aug 21 14:40 sql
drwxr-xr-x 6 hadoop hadoop   73 Aug 21 14:40 web

目录

描述

bin

存放启动/停止azkaban-solo-server的脚本

conf

azkaban-solo-server的配置文件

lib

依赖的jar包

extlib(解压后不存在)

依赖的jar包

plugins

插件的安装目录

web

azkaban-solo-server所有web相关的文件(CSS/图片/HTML等)

conf目录下有3个文件:

文件

描述

azkaban.properties

主要配置文件,配置运行时参数

azkaban-users.xml

权限控制文件,用于添加用户和角色

global.properties

全局配置,作用于所有的workflow和job

azkaban.private.properties(解压后不存在)

用于存放密码,例如MySQL的密码

  • 修改conf/azkaban.properties文件
# 把时区改为上海
default.timezone.id=Asia/Shanghai
  • 直接启动服务
[hadoop@beh07 azkaban-solo-server]$ bin/start-solo.sh

注意:启动服务需要在azkaban-solo-server目录下执行,假如你进入bin目录下执行./start-solo.sh,那么由于配置文件中默认使用的是相对路径,可能会发生找不到文件的错误。

启动后新创建了一些目录:

drwxr-xr-x 4 hadoop hadoop   126 Aug 21 16:34 bin
drwxr-xr-x 2 hadoop hadoop    82 Aug 21 15:55 conf
-rw-rw-r-- 1 hadoop hadoop     6 Aug 21 16:37 currentpid
drwxrwsr-x 2 hadoop hadoop     6 Aug 21 16:37 executions
-rw-rw-r-- 1 hadoop hadoop     5 Aug 21 16:37 executor.port
-rw-rw-r-- 1 hadoop hadoop 45056 Aug 21 16:37 h2.mv.db
drwxr-xr-x 2 hadoop hadoop  4096 Aug 21 14:40 lib
drwxrwxr-x 2 hadoop hadoop    35 Aug 21 16:37 local
drwxr-xr-x 3 hadoop hadoop    22 Aug 21 14:40 plugins
drwxrwxr-x 2 hadoop hadoop     6 Aug 21 16:37 projects
-rw-rw-r-- 1 hadoop hadoop 15892 Aug 21 16:37 soloServerLog__2019-08-21+16:37:47.out
drwxr-xr-x 2 hadoop hadoop  4096 Aug 21 16:37 sql
drwxrwxr-x 2 hadoop hadoop     6 Aug 21 16:37 temp
drwxr-xr-x 6 hadoop hadoop    73 Aug 21 14:40 web
  • 访问WEB-UI

账号密码是azkaban,登陆后页面如下:

这样,Azkaban solo-server模式就已经搭建成功了!

需要停止服务的话,执行以下命令即可:

[hadoop@beh07 azkaban-solo-server]$ bin/shutdown-solo.sh
Killing solo-server. [pid: 16035], attempt: 1
shutdown succeeded

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

扫码关注云+社区

领取腾讯云代金券