首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >storm(2)-基于k8s的安装

storm(2)-基于k8s的安装

作者头像
yiduwangkai
发布2019-09-17 15:48:33
1K0
发布2019-09-17 15:48:33
举报
文章被收录于专栏:大数据进阶大数据进阶

上一篇是写的storm基于物理机的安装

首先我们先来编写一下Dockerfile

FROM ip:80/bigdata/centos7-jdk:1.8
 #config storm env
 ENV STORM_USER=storm \
     STORM_INSTALL_BASE=/home/frank/platform \
     STORM_HOME=/home/frank/platform/storm \
     STORM_CONF_DIR=/home/frank/conf/storm \
     STORM_DIR=/home/frank/hard_disk/storm \
     STORM_BLOBS=/home/frank/hard_disk/storm/blobs \
     STORM_NIMBUS=/home/frank/hard_disk/storm/nimbus \
     STORM_LOG_DIR=/home/frank/logs/storm \ 
     STORM_ROOT=/home/frank \
     STORM_LOG=/home/frank/logs
 ENV PATH $PATH:$STORM_HOME/bin
 #add FLINK user and make dirs
 RUN set -x \
     && adduser "$STORM_USER" \
     && mkdir -p "$STORM_INSTALL_BASE" "$STORM_CONF_DIR" "$STORM_LOG" "$STORM_LOG_DIR" "$STORM_BLOBS" "$STORM_NIMBUS"
 #install storm libs
 ADD apache-storm-1.2.2.tar.gz  $STORM_INSTALL_BASE
 RUN cd $STORM_INSTALL_BASE && ln -s -f apache-storm-1.2.2 storm && mkdir -p /home/storm
RUN chown -R "$STORM_USER:$STORM_USER" "$STORM_ROOT"
 COPY docker-entrypoint.sh /home/storm
 RUN chmod 755 /home/storm/docker-entrypoint.sh
 USER storm
 ENTRYPOINT ["/home/storm/docker-entrypoint.sh"]

其中的脚本如下:

#!/bin/sh
 # If unspecified, the hostname of the container is taken as the JobManager address
 ACTION_CMD="$1"
 source /etc/profile 
 export STORM_CONF_DIR=/home/frank/conf/storm
 export STORM_CONF_FILE=/home/frank/conf/storm/storm.yaml
 sed -i -e "s/storm.zookeeper.servers: \[localhost\]/storm.zookeeper.servers: \[${STORM_ZK_QUORUM}\]/" /home/frank/conf/storm/storm.yaml
 sed -i -e "s/storm.zookeeper.root: \"\/storm\"/storm.zookeeper.root: \/${STORM_ZK_ROOT}/" /home/frank/conf/storm/storm.yaml
 # if use cluster model, pod ${JOB_CLUSTER_NAME}-0,${JOB_CLUSTER_NAME}-1 as jobmanager
 if [ ${ACTION_CMD} == "cluster" ]; then
   nimbuses=(${NIMBUS_HOSTS//,/ })
   ACTION_CMD="supervisor"
   for i in ${!nimbuses[@]}
   do
       if [ "$(hostname -s)" == "${nimbuses[i]}" ]; then
           NIMBUS_ADDRESS=${NIMBUS_ADDRESS:-$(hostname -f)}
           ACTION_CMD="nimbus"
           sed -i -e "s/nimbus.seeds: \[localhost\]/nimbus.seeds: \["${NIMBUS_ADDRESS}"\]/" /home/frank/conf/storm/storm.yaml
           echo "pod hostname match nimbus config host, change action to nimbus."
       fi
   done
 fi
 # if ha model, replace ha configuration
 if [ ${ACTION_CMD} == "help" ]; then
     echo "Usage: $(basename "$0") (cluster|nimbus|supervisor|ui|logviewer|help)"
     exit 0
 elif [ ${ACTION_CMD} == "nimbus" ]; then
     echo "Starting nimbus ui logviewer supervisor"
     nohup /home/frank/platform/storm/bin/storm nimbus >> /tmp/nimbus.log  2>&1 & \
     nohup /home/frank/platform/storm/bin/storm ui >> /tmp/ui.log 2>&1 & \
     /home/frank/platform/storm/bin/storm logviewer
 elif [ ${ACTION_CMD} == "supervisor" ]; then
     echo "Starting supervisor"
     /home/frank/platform/storm/bin/storm supervisor 
 fi

镜像构建命令如下:

docker build --network=host -f Dockerfile -t ip:80/bigdata/storm:1.2.2 .

下面我们提供一下service的创建文件

apiVersion: v1
 kind: Service
 metadata:
   name: storm-cluster
   labels:
     app: storm-cluster
   namespace: storm
 spec:
   clusterIP: None
   ports:
     - port: 8080
       name: ui
   selector:
     app: storm-cluster
apiVersion: apps/v1
 kind: StatefulSet
 metadata:
   name: storm-cluster
   namespace: storm
 spec:
   selector:
     matchLabels:
       app: storm-cluster
   serviceName: storm-cluster
   replicas: 4
   podManagementPolicy: Parallel
   template:
     metadata:
       labels:
         app: storm-cluster
     spec:
       terminationGracePeriodSeconds: 2
       containers:
       - name: storm-cluster
         imagePullPolicy: Always
         image: ip:80/bigdata/storm:1.2.2
         command: ["/bin/sleep"]
         args: ["30d"]
         volumeMounts:
           - name: storm-conf
             mountPath: /home/conf/storm
           - name: storm-log
             mountPath: /home/logs/storm
         resources:
           requests:
             memory: "4096Mi"
             cpu: 4
           limits:
             memory: "4096Mi"
             cpu: 4
         env:
         - name: NIMBUS_HOSTS
           value: "storm-cluster-2,storm-cluster-3"
         - name: STORM_ZK_QUORUM
           value: "ip"
         - name: STORM_ZK_ROOT
           value: "stormNew"
         ports:
         - containerPort: 6627
           name: nimbus
         - containerPort: 8000
           name: logviewer
         - containerPort: 8080
           name: storm-ui
         - containerPort: 6700
           name: sv0
         - containerPort: 6701
           name: sv1
         - containerPort: 6702
           name: sv2
         - containerPort: 6703
           name: sv3
         - containerPort: 22
           name: ssh
       volumes:
         - name: storm-conf
           configMap:
             name: storm-conf
         - name: storm-log
           hostPath:
             path: /stormtmp
             type: Directory
             
 service的创建脚本如下:
 

kubectl create -f storm-svc.yaml

statefulSet的创建脚本如下:

kubectl create -f storm-statefulSet.yaml

这个创建脚本要注意如下问题:

command:/bin/sleep

程序需要手动启动

/home/vipshop/hard_disk/storm
drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 blobs
 drwxr-xr-x 2 storm storm 4096 Jul 12 12:20 nimbus
 drwxrwxr-x 5 storm storm 4096 Jul 29 00:00 supervisor
 这个hard_disk目录需要有storm权限

注:nimbus.thrift.max_buffer_size: 10485760

这个参数需要是2的倍数

本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
相关产品与服务
容器镜像服务
容器镜像服务(Tencent Container Registry,TCR)为您提供安全独享、高性能的容器镜像托管分发服务。您可同时在全球多个地域创建独享实例,以实现容器镜像的就近拉取,降低拉取时间,节约带宽成本。TCR 提供细颗粒度的权限管理及访问控制,保障您的数据安全。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档