008.Kafka集群管理工具CMAK-3.0.0.5源码编译以及安装部署

# 1.安装
[admin@bdc01 ~]$ sudo wget -O /etc/yum.repos.d/bintray-sbt-rpm.repo https://bintray.com/sbt/rpm/rpm
[admin@bdc01 ~]$ sudo yum install sbt -y

# 2.配置，文件：/etc/sbt/sbtopts，未列出的保持默认即可
-sbt-create
-sbt-dir /opt/sbt
-sbt-boot /opt/sbt/boot
-ivy /data/sbt/repo
-Dsbt.override.build.repos=true
-Dsbt.repository.config=/etc/sbt/repositories
-Dsbt.repository.secure=false
-Dfile.encoding=UTF8
-Dsbt.log.format=true

# 3.创建自定义的目录
[admin@bdc01 ~]$ sudo mkdir -p /opt/sbt/boot
[admin@bdc01 ~]$ sudo mkdir -p /data/sbt/repo
[admin@bdc01 ~]$ sudo chown -R admin:admin /opt/sbt
[admin@bdc01 ~]$ sudo chown -R admin:admin /data/sbt

# 4.优化仓库地址，加速下载，文件：/etc/sbt/repositories
[repositories]
local
public: https://maven.aliyun.com/nexus/content/groups/public/
typesafe: https://dl.bintray.com/typesafe/ivy-releases/ , [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
ivy-sbt-plugin: https://dl.bintray.com/sbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
sonatype-oss-releases
sonatype-oss-snapshots

# 5.下载jar包
[admin@bdc01 ~]$ sbt sbtVersion
# 到这一行会卡住一会，耐心等待，之后下载会很快
[info] [launcher] getting org.scala-sbt sbt 1.4.2  (this may take some time)...
downloading https://maven.aliyun.com/nexus/content/groups/public/org/scala-sbt/sbt/1.4.2/sbt-1.4.2.jar ...
downloading https://maven.aliyun.com/nexus/content/groups/public/org/scala-lang/scala-library/2.12.12/scala-library-2.12.12.jar ...
    [SUCCESSFUL ] org.scala-sbt#sbt;1.4.2!sbt.jar (528ms)
    ...
[info] welcome to sbt 1.4.2 (Oracle Corporation Java 1.8.0_251)
...
[info] Fetched artifacts of 
[info] set current project to admin (in build file:/home/admin/)
[info] 1.4.2

# 1.下载源码，解压后进入源码根目录
[admin@bdc01 ~]$ wget https://github.com/yahoo/CMAK/archive/3.0.0.5.tar.gz
[admin@bdc01 ~]$ tar -zxvf 3.0.0.5.tar.gz
[admin@bdc01 ~]$ cd CMAK-3.0.0.5/

# 2.编译源码
# 注意：是sbt clean dist，不是./sbt clean dist
[admin@bdc01 CMAK-3.0.0.5]$ sbt clean dist

# 我在执行过程中遇到以下报错
[error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$WrongChecksum: wrong checksum: /home/admin/.cache/coursier/v1/https/maven.aliyun.com/nexus/content/groups/public/org/webjars/npm/entities/2.1.0/entities-2.1.0.jar (expected SHA-1 1bcaa0b8032c97dbad046323af72ff0213ebcac5 in /home/admin/.cache/coursier/v1/https/maven.aliyun.com/nexus/content/groups/public/org/webjars/npm/entities/2.1.0/entities-2.1.0.jar.sha1, got c36a3cc098a65d94b215834942e806fc8d1af23a)
[error]     at lmcoursier.internal.shaded.coursier.cache.FileCache.$anonfun$validateChecksum$4(FileCache.scala:705)
[error]     at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error]     at scala.util.Success.$anonfun$map$1(Try.scala:255)
...

# 这里是说，org/webjars/npm/entities/2.1.0/entities-2.1.0.jar这个jar包
# 目录下有个entities-2.1.0.jar.sha1文件，其中记录的sha1值为1bcaa0b8032c97dbad046323af72ff0213ebcac5
# 而通过sha1sum计算出来的值是c36a3cc098a65d94b215834942e806fc8d1af23a
# 这说明这个文件有问题
# 我验证了一下，果然sha1值对不上
# 于是我在maven的中央仓库下载了该jar包，https://repo1.maven.org/maven2/org/webjars/npm/entities/2.1.0/entities-2.1.0.jar
# 使用sha1sum计算，得到的值是c36a3cc098a65d94b215834942e806fc8d1af23a
# 说明，阿里云仓库下载下来的jar包与maven中央仓库下载的jar包是一样的
# 但是不知道什么原因，entities-2.1.0.jar.sha1文件中的值有问题
# 于是我把entities-2.1.0.jar.sha1文件中的值改成了c36a3cc098a65d94b215834942e806fc8d1af23a
# 再执行，就没有问题了

# 依赖的jar包下完之后，报了一推错
[error] /home/admin/CMAK-3.0.0.5/app/kafka/manager/actor/cluster/KafkaStateActor.scala:115:29: type mismatch;
[error]  found   : java.util.Map[_$1,_$2] where type _$2, type _$1
[error]  required: java.util.Map[? <: Object, ? <: Object]
[error] Note: _$1 >: ? <: Object, but Java-defined trait Map is invariant in type K.
[error] You may wish to investigate a wildcard type such as `_ >: ? <: Object`. (SLS 3.2.10)
[error] Note: _$2 >: ? <: Object, but Java-defined trait Map is invariant in type V.
[error] You may wish to investigate a wildcard type such as `_ >: ? <: Object`. (SLS 3.2.10)
[error]       cp => props.putAll(cp.asMap)
[error]                             ^
[error] /home/admin/CMAK-3.0.0.5/app/kafka/manager/actor/cluster/KafkaStateActor.scala:275:29: type mismatch;
[error]  found   : java.util.Map[_$1,_$2] where type _$2, type _$1
[error]  required: java.util.Map[? <: Object, ? <: Object]
[error] Note: _$1 >: ? <: Object, but Java-defined trait Map is invariant in type K.
[error] You may wish to investigate a wildcard type such as `_ >: ? <: Object`. (SLS 3.2.10)
[error] Note: _$2 >: ? <: Object, but Java-defined trait Map is invariant in type V.
[error] You may wish to investigate a wildcard type such as `_ >: ? <: Object`. (SLS 3.2.10)
[error]       cp => props.putAll(cp.asMap)
...
# 我的环境是JDK1.8，由于官网介绍cmak都是使用JDK11编译的于是我尝试使用JDK11再进行一次编译，这次编译成功了
[admin@bdc01 CMAK-3.0.0.5]$ sbt -java-home /usr/java/jdk-11.0.9 clean dist
copying runtime jar...
[info] welcome to sbt 1.4.2 (Oracle Corporation Java 11.0.9)
[info] loading project definition from /home/admin/CMAK-3.0.0.5/project/project
[info] loading settings for project cmak-3-0-0-5-build from plugins.sbt ...
[info] loading project definition from /home/admin/CMAK-3.0.0.5/project
[info] loading settings for project root from build.sbt ...
[info] set current project to cmak (in build file:/home/admin/CMAK-3.0.0.5/)
[success] Total time: 0 s, completed Nov 6, 2020, 2:20:26 PM
[info] Wrote /home/admin/CMAK-3.0.0.5/target/scala-2.12/cmak_2.12-3.0.0.5.pom
Warning: node.js detection failed, sbt will use the Rhino based Trireme JavaScript engine instead to run JavaScript assets compilation, which in some cases may be orders of magnitude slower than using node.js.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Main Scala API documentation to /home/admin/CMAK-3.0.0.5/target/scala-2.12/api...
[info] Non-compiled module 'compiler-bridge_2.12' for Scala 2.12.10. Compiling...
[info] compiling 136 Scala sources and 2 Java sources to /home/admin/CMAK-3.0.0.5/target/scala-2.12/classes ...
[info]   Compilation completed in 11.645s.
model contains 632 documentable templates
[info] Main Scala API documentation successful.
[info] LESS compiling on 1 source(s)
[success] All package validations passed
[info] Your package is ready in /home/admin/CMAK-3.0.0.5/target/universal/cmak-3.0.0.5.zip
[success] Total time: 82 s (01:22), completed Nov 6, 2020, 2:21:48 PM

# 在CMAK-3.0.0.5/target/universal目录下，有一个文件：cmak-3.0.0.5.zip
# 就是编译好的安装包

# 1.安装JDK11
# cmak都是使用JDK11版本来编译的，所以使用JDK8是无法运行的
# 所以给cmak单独部署JDK11环境
[admin@bdc01 java]$ ll /usr/java/
total 0
lrwxrwxrwx 1 root root  12 2020-06-10 14:16 jdk -> jdk1.8.0_251
drwxr-xr-x 8 root root 115 2020-11-06 14:17 jdk-11.0.9
drwxr-xr-x 7 root root 245 2020-03-12 14:37 jdk1.8.0_251
# 我的JAVA_HOME实际上为JDK1.8
[admin@bdc01 java]$ echo $JAVA_HOME
/usr/java/jdk

# 2.解压
[admin@bdc01 opt]$ sudo unzip cmak-3.0.0.5.zip
[admin@bdc01 opt]$ sudo chown -R admin:admin cmak-3.0.0.5

# 3.修改配置文件
# 文件：cmak-3.0.0.5/conf/application.conf
application.home=/opt/cmak-3.0.0.5
cmak.zkhosts="bdc01:2181,bdc02:2181,bdc03:2181"
basicAuthentication.enabled=true
basicAuthentication.username="admin"
basicAuthentication.password="admin"
# 文件cmak-3.0.0.5/conf/logback.xml、cmak-3.0.0.5/conf/logger.xml
# 在根标签<configuration>下添加如下配置
<property name="application.home" value="/opt/cmak-3.0.0.5"/>

# 4.Kafka开启JMX端口
# 文件：kafka/bin/kafka-run-class.sh
# 在文件开头加入以下环境变量
export JMX_PORT=9988
# 同步配置后重启Kafka集群

# 5.启动cmak
[admin@bdc01 cmak-3.0.0.5]$ nohup /opt/cmak-3.0.0.5/bin/cmak -Dhttp.port=9998 -java-home /usr/java/jdk-11.0.9 > /dev/null &
[1] 47282

# 6.进入Zookeeper客户端，创建/kafka-manager/mutex节点
# 如果zookeeper的版本是>3.4的，则这一步是不需要的
[admin@bdc01 cmak-3.0.0.5]$ zkCli.sh        
[zk: localhost:2181(CONNECTED) 0] create /kafka-manager ""
Created /kafka-manager
[zk: localhost:2181(CONNECTED) 1] create /kafka-manager/mutex ""
Created /kafka-manager/mutex
[zk: localhost:2181(CONNECTED) 2] create /kafka-manager/mutex/locks ""
Created /kafka-manager/locks
[zk: localhost:2181(CONNECTED) 3] create /kafka-manager/mutex/leases ""
Created /kafka-manager/leases
[zk: localhost:2181(CONNECTED) 4] quit