K8S容器环境下GitLab-CI和GItLab Runner 部署记录

洗尽了浮华

发布于 2021-04-01 10:25:58

6.6K0

发布于 2021-04-01 10:25:58

文章被收录于专栏：散尽浮华散尽浮华

一、简单介绍

GitLab-CI

GitLab CI/CD是GitLab的一部分，支持从计划到部署具有出色的用户体验。CI/CD是开源GitLab社区版和专有GitLab企业版的一部分。可以根据需要添加任意数量的计算节点，每个构建可以拆分为多个作业，这些作业可以在多台计算机上并行运行。
GitLab-CI轻量级，不需要复杂的安装手段。配置简单，与gitlab可直接适配。实时构建日志十分清晰，UI交互体验很好。使用 YAML 进行配置，任何人都可以很方便的使用。GitLabCI 有助于DevOps人员，例如敏捷开发中，开发与运维是同一个人，最便捷的开发方式。
在大多数情况，构建项目都会占用大量的系统资源，如果让gitlab本身来运行构建任务的话，显然Gitlab的性能会大幅度下降。GitLab-CI最大的作用就是管理各个项目的构建状态。因此，运行构建任务这种浪费资源的事情交给一个独立的Gitlab Runner来做就会好很多，更重要的是Gitlab Runner 可以安装到不同的机器上，甚至是我们本机，这样完全就不会影响Gitlab本身了。
从GitLab8.0开始，GitLab-CI就已经集成在GitLab中，我们只需要在项目中添加一个.gitlab-ci.yaml文件，然后运行一个Runner，即可进行持续集成。

GitLab-CI：集成、开源、无缝、可扩展、更快的结果、针对交付进行了优化：

GItLab Runner

Gitlab Runner是一个开源项目，用于运行您的作业并将结果发送给gitlab。它与Gitlab CI结合使用，gitlab ci是Gitlab随附的用于协调作用的开源持续集成服务。
Gitlab Runner是用Go编写的，可以作为一个二进制文件运行，不需要特定于语言的要求
它皆在GNU/Linux，MacOS和Windows操作系统上运行。另外注意：如果要使用Docker，Gitlab Runner要求Docker 至少是v1.13.0版本才可以。

Kubernetes Gitlab CICD 演示图：

二、基于Kubernetes Gitlab CICD 容器化部署记录

Gitlab官方提供了Helm的方式在Kubernetes集群中来快速安装，但是在使用的过程中发现Helm提供的Chart包中有很多其他额外的配置。所以这里我采用K8S自定义的方式来安装。
Gitlab主要涉及3个应用：Redis、Postgresql、Gitlab核心程序。
本案例中使用的Gitlab-ce镜像部署，镜像中的Gitlab版本是13.7.4。
本案例中使用NFS作为持久化存储方式。除此之外，还可以选择HostPath本地持久化存储、NAS云端持久化存储、Ceph分布式持久化存储等。

注意：本示例部署所涉及到的image镜像均导入到Harbor私有私仓（172.16.60.230）。

1）使用NFS作为持久化存储

在NFS服务器端（172.16.60.238）创建Redis、Postgresql、Gitlab核心程序容器的持久化挂载目录

[root@k8s-harbor01 ~]# mkdir -p /data/storage/k8s/gitlab/{postgresql,redis,gitlab}
[root@k8s-harbor01 ~]# ll /data/storage/k8s/gitlab/
total 0
drwxr-xr-x 2 root root 6 Mar 25 14:03 gitlab
drwxr-xr-x 2 root root 6 Mar 25 14:03 postgresql
drwxr-xr-x 2 root root 6 Mar 25 14:03 redis

2）部署Gitlab

可以先创建一个命名空间

[root@k8s-master01 gitlab]# kubectl create ns kube-ops

[root@k8s-master01 gitlab]# kubectl get ns|grep kube-ops
kube-ops              Active   7d18h

配置三个核心程序的容器化部署的yaml文件

[root@k8s-master01 gitlab]# pwd
/opt/k8s/k8s_project/gitlab
[root@k8s-master01 gitlab]# ll
total 12
-rw-r--r-- 1 root root 1629 Mar 25 14:05 gitlab-postgresql.yaml
-rw-r--r-- 1 root root 1207 Mar 25 14:05 gitlab-redis.yaml
-rw-r--r-- 1 root root 2691 Mar 25 14:05 gitlab.yaml

gitlab-postgresql.yaml 文件内容：

[root@k8s-master01 gitlab]# cat gitlab-postgresql.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgresql
  namespace: kube-ops
  labels:
    name: postgresql
spec:
  replicas: 1
  selector:
    matchLabels:
      name: postgresql
  template:
    metadata:
      name: postgresql
      labels:
        name: postgresql
    spec:
      containers:
      - name: postgresql
        image: 172.16.60.230/gitlab/postgresql:v1
        imagePullPolicy: IfNotPresent
        env:
        - name: DB_USER
          value: gitlab
        - name: DB_PASS
          value: passw0rd
        - name: DB_NAME
          value: gitlab_production
        - name: DB_EXTENSION
          value: pg_trgm
        ports:
        - name: postgres
          containerPort: 5432
        volumeMounts:
        - mountPath: /var/lib/postgresql
          name: data
        livenessProbe:
          exec:
            command:
            - pg_isready
            - -h
            - localhost
            - -U
            - postgres
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - pg_isready
            - -h
            - localhost
            - -U
            - postgres
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        nfs:
          server: 172.16.60.238
          path: /data/storage/k8s/gitlab/postgresql
          readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  name: postgresql
  namespace: kube-ops
  labels:
    name: postgresql
spec:
  ports:
    - name: postgres
      port: 5432
      targetPort: postgres
  selector:
    name: postgresql

gitlab-redis.yaml 文件内容：

[root@k8s-master01 gitlab]# cat gitlab-redis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: kube-ops
  labels:
    name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      name: redis
  template:
    metadata:
      name: redis
      labels:
        name: redis
    spec:
      containers:
      - name: redis
        image: 172.16.60.230/gitlab/redis:latest
        imagePullPolicy: IfNotPresent
        ports:
        - name: redis
          containerPort: 6379
        volumeMounts:
        - mountPath: /var/lib/redis
          name: data
        livenessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        nfs:
          server: 172.16.60.238
          path: /data/storage/k8s/gitlab/redis
          readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: kube-ops
  labels:
    name: redis
spec:
  ports:
    - name: redis
      port: 6379
      targetPort: redis
  selector:
    name: redis

gitlab.yaml 文件内容：

今天的极致任务：
把所有的面试题收集完成
全力补到Python


[root@k8s-master01 gitlab]# cat gitlab.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: kube-ops
  labels:
    name: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab
  template:
    metadata:
      name: gitlab
      labels:
        name: gitlab
    spec:
      containers:
      - name: gitlab
        image: 172.16.60.230/gitlab/gitlab-ce:latest
        imagePullPolicy: IfNotPresent
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: GITLAB_TIMEZONE
          value: Beijing
        - name: GITLAB_SECRETS_DB_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_SECRETS_SECRET_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_SECRETS_OTP_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_ROOT_PASSWORD
          value: admin321
        - name: GITLAB_ROOT_EMAIL
          value: 1025337607@qq.com
        - name: GITLAB_HOST
          value: 0.0.0.0:30004
        - name: GITLAB_PORT
          value: "80"
        - name: GITLAB_SSH_PORT
          value: "22"
        - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
          value: "true"
        - name: GITLAB_NOTIFY_PUSHER
          value: "false"
        - name: GITLAB_BACKUP_SCHEDULE
          value: daily
        - name: GITLAB_BACKUP_TIME
          value: 01:00
        - name: DB_TYPE
          value: postgres
        - name: DB_HOST
          value: postgresql
        - name: DB_PORT
          value: "5432"
        - name: DB_USER
          value: gitlab
        - name: DB_PASS
          value: passw0rd
        - name: DB_NAME
          value: gitlab_production
        - name: REDIS_HOST
          value: redis
        - name: REDIS_PORT
          value: "6379"
        ports:
        - name: http
          containerPort: 80
        - name: ssh
          containerPort: 22
        volumeMounts:
        - mountPath: /home/git/data
          name: data
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 180
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        nfs:
          server: 172.16.60.238
          path: /data/storage/k8s/gitlab/gitlab
          readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  name: gitlab
  namespace: kube-ops
  labels:
    name: gitlab
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: http
      nodePort: 30004
    - name: ssh
      port: 22
      targetPort: ssh
  selector:
    name: gitlab

创建并启动gitlab相关容器进程：

[root@k8s-master01 gitlab]# ll
total 12
-rw-r--r-- 1 root root 1629 Mar 25 14:05 gitlab-postgresql.yaml
-rw-r--r-- 1 root root 1207 Mar 25 14:05 gitlab-redis.yaml
-rw-r--r-- 1 root root 2691 Mar 25 14:05 gitlab.yaml

[root@k8s-master01 gitlab]# kubectl apply -f .
deployment.apps/postgresql created
service/postgresql created
deployment.apps/redis created
service/redis created
deployment.apps/gitlab created
service/gitlab created

稍微等一会儿（由于程序启动顺序原因，pod可能会出现重启次数，不过最终都会启动成功），

查看pod状态：

[root@k8s-master01 gitlab]# kubectl get pods -n kube-ops
NAME                          READY   STATUS    RESTARTS   AGE
gitlab-5b887894d5-ntxzj       1/1     Running   1          38m
postgresql-57bf98cdf8-7mdh9   1/1     Running   1          38m
redis-56769dc6b6-c4rnq        1/1     Running   0          38m

查看svc：

[root@k8s-master01 gitlab]# kubectl get svc -n kube-ops
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                     AGE
gitlab       NodePort    10.254.48.72   <none>        80:30004/TCP,22:32280/TCP   14m
postgresql   ClusterIP   10.254.88.39   <none>        5432/TCP                    14m
redis        ClusterIP   10.254.198.0   <none>        6379/TCP

3）访问 Gitlab

这里采用NodePort的方式，通过 http://任意node节点ip:30004 地址访问Gitlab
Gitlab登录用户名：root，密码：admin321
Gitlab登录密码可以在yaml文件里修改

这里容器化部署后的Gitlab版本是13.7.4

4）创建演示项目

接下来顺便创建一个项目，用于演示：

接下来在服务器上git clone，进行代码提交演示：

git clone地址是 http://gitlab-5b887894d5-ntxzj/root/kevin-test.git

地址中的gitlab-5b887894d5-ntxzj是pod名称，在容器外部访问不了，需要修改为对应的nodeport地址，故git clone地址可以是：http://172.16.60.234:30004/root/kevin-test.git

在其中一个node节点上进行代码提交演示：

[root@k8s-node02 mnt]# mkdir /mnt/haha
[root@k8s-node02 mnt]# cd /mnt/haha
[root@k8s-node02 haha]# git config --global user.email "1025337607@qq.com"
[root@k8s-node02 haha]# git config --global user.name "Administrator"
[root@k8s-node02 haha]#

[root@k8s-node02 haha]# git clone http://172.16.60.234:30004/root/kevin-test.git
Cloning into 'kevin-test'...
Username for 'http://172.16.60.234:30004': root            #输入账号root
Password for 'http://root@172.16.60.234:30004':            #输入账号root的密码
warning: You appear to have cloned an empty repository.
[root@k8s-node02 haha]# ls
kevin-test
[root@k8s-node02 haha]# cd kevin-test/
[root@k8s-node02 kevin-test]# ll
total 0
[root@k8s-node02 kevin-test]#

[root@k8s-node02 kevin-test]# touch test.md
[root@k8s-node02 kevin-test]# echo "come on" > test.md
[root@k8s-node02 kevin-test]# git add test.md
[root@k8s-node02 kevin-test]# git commit -m "add test.md"
[master (root-commit) 8ccda29] add test.md
 1 file changed, 1 insertion(+)
 create mode 100644 test.md

 [root@k8s-node02 kevin-test]# git commit -m "add test.md"
[master (root-commit) 8ccda29] add test.md
 1 file changed, 1 insertion(+)
 create mode 100644 test.md
[root@k8s-node02 kevin-test]# git push -u origin master
Username for 'http://172.16.60.234:30004': root
Password for 'http://root@172.16.60.234:30004':
Counting objects: 3, done.
Writing objects: 100% (3/3), 216 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To http://172.16.60.234:30004/root/kevin-test.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.

如何解决 "每次输入用户名和密码" 的问题？

在代码目录.git/config文件内[remote "origin"]的url的gitlab域名前添加gitlab注册时的"用户名:密码@"

[root@k8s-node02 kevin-test]# cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = http://root:admin321@172.16.60.234:30004/root/kevin-test.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master

接着再次尝试提交内容，就不需要手动输入用户名和密码了：

[root@k8s-node02 kevin-test]# git pull
Already up-to-date.

[root@k8s-node02 kevin-test]# echo "this is gitlab test" > test.md
[root@k8s-node02 kevin-test]# git add test.md
[root@k8s-node02 kevin-test]# git commit -m "modified test.md"
[master f2fbb27] modified test.md
 1 file changed, 1 insertion(+), 1 deletion(-)
 
[root@k8s-node02 kevin-test]# git push -u origin master
Counting objects: 12, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (10/10), 845 bytes | 0 bytes/s, done.
Total 10 (delta 1), reused 0 (delta 0)
To http://root:admin321@172.16.60.234:30004/root/kevin-test.git
   fe40316..f2fbb27  master -> master
Branch master set up to track remote branch master from origin.

代码上传后，gitlab上展示效果如下：

进入该项目下，左侧栏CICD里有三种方式：Pipelines、Jobs、Schedules

5）Gitlab Runner 安装和注册

gitlab runner支持多种方式安装，我这里就采取在k8s中安装。

官方文档地址: https://docs.gitlab.com/runner/install/

打开gitlab，如下图所示，左边代表runner状态，右边是配置runner信息。

注意右边栏的token信息，后面注册runner的时候会用到：

接下来进行配置gitlab runner资源清单 (runner-configmap.yaml)

[root@k8s-master01 gitlab]# cat runner-configmap.yaml
apiVersion: v1
data:
  REGISTER_NON_INTERACTIVE: "true"
  REGISTER_LOCKED: "false"
  METRICS_SERVER: "0.0.0.0:9100"
  CI_SERVER_URL: "http://gitlab.kube-ops.svc.cluster.local/ci"
  RUNNER_REQUEST_CONCURRENCY: "4"
  RUNNER_EXECUTOR: "kubernetes"
  KUBERNETES_NAMESPACE: "kube-ops"
  KUBERNETES_PRIVILEGED: "true"
  KUBERNETES_CPU_LIMIT: "1"
  KUBERNETES_CPU_REQUEST: "500m"
  KUBERNETES_MEMORY_LIMIT: "1Gi"
  KUBERNETES_SERVICE_CPU_LIMIT: "1"
  KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
  KUBERNETES_HELPER_CPU_LIMIT: "500m"
  KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
  KUBERNETES_PULL_POLICY: "if-not-present"
  KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
  KUBERNETES_POLL_INTERVAL: "5"
  KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-cm
  namespace: kube-ops

需要注意：

CI_SERVER_URL 这个地址是gitlab的地址，如果gitlab在宿主机直接写宿主机的ip即可，容器是格式为：svc名称.命名空间.svc.cluster.local (如果都按照我的文档来进行安装不需要修改别的配置了)。
如果定义的gitlab域名并不是通过外网DNS解析，而是通过/etc/hosts进行映射，那么我们需要在Runner的Pod中去添加对应的hosts，需要通过--pre-clone-script参数来指定一段脚本来添加hosts信息，也就是在ConfigMap中添加环境变量RUNNER_PRE_CLONE_SCRIPT的值：

本案例，这里gitlab地址我是使用node节点的ip+port方式。如果使用gitlab域名方式，且不是外网DNS解析，比如域名地址是http://gitlab.kevin.com 则需要在上面的ConfigMap中添加环境变量RUNNER_PRE_CLONE_SCRIPT的值: RUNNER_PRE_CLONE_SCRIPT = "echo 'xx.xx.xxx.xx git.i4t.com' >> /etc/hosts" 其中xx.xx.xxx.xx 为node节点ip地址

另外记住：在ConfigMap添加新选项后，需要删除Gitlab ci Runner Pod

因为这里我是使用envFrom来注入上面的这些环境变量而不是直接使用env(envfrom 通过将环境变量放置到ConfigMaps或Secrets来帮助减小清单文件)

如果我们想添加其他选项，那么可以在等到后面的gitlab-ci-runner的Pod容器启动成功后，登录gitlab-ci-runner的pod容器内部运行gitlab-ci-multi-runner register --help 命令来查看所有可使用的选项，只需要为配置的标志添加env变量即可：

gitlab-runner@gitlab-ci-runner-0:/$ gitlab-ci-multi-runner register --help
[...]
--kubernetes-cpu-limit value                          The CPU allocation given to build containers (default: "1") [$KUBERNETES_CPU_LIMIT]
--kubernetes-memory-limit value                       The amount of memory allocated to build containers (default: "4Gi") [$KUBERNETES_MEMORY_LIMIT]
--kubernetes-service-cpu-limit value                  The CPU allocation given to build service containers (default: "1") [$KUBERNETES_SERVICE_CPU_LIMIT]
--kubernetes-service-memory-limit value               The amount of memory allocated to build service containers (default: "1Gi") [$KUBERNETES_SERVICE_MEMORY_LIMIT]
--kubernetes-helper-cpu-limit value                   The CPU allocation given to build helper containers (default: "500m") [$KUBERNETES_HELPER_CPU_LIMIT]
--kubernetes-helper-memory-limit value                The amount of memory allocated to build helper containers (default: "3Gi") [$KUBERNETES_HELPER_MEMORY_LIMIT]
--kubernetes-cpu-request value                        The CPU allocation requested for build containers [$KUBERNETES_CPU_REQUEST]
...
--pre-clone-script value                              Runner-specific command script executed before code is pulled [$RUNNER_PRE_CLONE_SCRIPT]
[...]

创建资源清单的configmap

[root@k8s-master01 gitlab]# kubectl apply -f runner-configmap.yaml
configmap/gitlab-ci-runner-cm created
[root@k8s-master01 gitlab]# kubectl get configmaps  -n kube-ops
NAME                  DATA   AGE
gitlab-ci-runner-cm   19     4s

可通过下面命令来查看此configmap内容：
[root@k8s-master01 gitlab]# kubectl describe cm gitlab-ci-runner-cm -n kube-ops

此时，还需要配置一个用于注册、运行和取消gitlab ci runner的小脚本。只有当Pod正常通过K8S (TERM信号)的终止流程时，才会触发注销注册。如果强行终止Pod（SIGKILL信号），Runner将不会自己注销自身。必须手动完成对这种Runner的清理 (注意：只有如这里在k8s集群里安装GitLan Runner才这样操作，二进制安装非K8s上安装则不受这个影响)

[root@k8s-master01 gitlab]# cat runner-scripts-cm.yaml
apiVersion: v1
data:
  run.sh: |
    #!/bin/bash
    unregister() {
        kill %1
        echo "Unregistering runner ${RUNNER_NAME} ..."
        /usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
        exit $?
    }
    trap 'unregister' EXIT HUP INT QUIT PIPE TERM
    echo "Registering runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
    sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /home/gitlab-runner/.gitlab-runner/config.toml
    echo "Starting runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
    wait
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-scripts
  namespace: kube-ops

创建此脚本的configmap

[root@k8s-master01 gitlab]# kubectl apply -f runner-scripts-cm.yaml
configmap/gitlab-ci-runner-scripts created

[root@k8s-master01 gitlab]# kubectl get configmaps  -n kube-ops
NAME                       DATA   AGE
gitlab-ci-runner-cm        19     4m12s
gitlab-ci-runner-scripts   1      11s

接着需要创建一个GITLAB_CI_TOKEN，然后使用gitlab ci runner token来创建一个Kubernetes secret对象。需要提前对token进行base64转码：

[root@k8s-master01 gitlab]# echo xWgpgrP3rSXnxvZL9oRf|base64 -w0
eFdncGdyUDNyU1hueHZaTDlvUmYK

特意注意：这里的token就是我们gitlab runner上截图的地方,base64只有在k8s环境上需要！

登录Gitlab，Runner右边栏token信息如下图：

使用上面的token创建一个Sercret对象 (gitlab-ci-token-secret.yaml)

[root@k8s-master01 gitlab]# cat gitlab-ci-token-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-ci-token
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
data:
  GITLAB_CI_TOKEN: eFdncGdyUDNyU1hueHZaTDlvUmYK

创建这个Secret

[root@k8s-master01 gitlab]# kubectl apply -f gitlab-ci-token-secret.yaml
secret/gitlab-ci-token created

[root@k8s-master01 gitlab]# kubectl get secret -n kube-ops|grep gitlab
gitlab-ci-token       Opaque                                1      19s

接下来创建真正运行Runner的控制器镜像，这里使用Statefulset，在开始运行的时候，尝试取消注册所有的同名Runner，当节点丢失时(即NodeLost事件)，这尤其有用，然后再尝试注册自己并开始运行。在正常停止Pod的时候，Runner将会运行unregister命令来尝试取消自己，所以gitlab就不能再使用这个Runner，这个则是通过kubernetes Pod生命周期中的hooks来完成的：

编译gitlab runner的pod部署的yaml文件内容：

[root@k8s-master01 gitlab]# cat runner-statefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: gitlab-ci-runner
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
spec:
  ports:
  - port: 9100
    targetPort: 9100
    name: http-metrics
  clusterIP: None
  selector:
    app: gitlab-ci-runner
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gitlab-ci-runner
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
spec:
  updateStrategy:
    type: RollingUpdate
  replicas: 2
  serviceName: gitlab-ci-runner
  selector:
    matchLabels:
      app: gitlab-ci-runner
  template:
    metadata:
      labels:
        app: gitlab-ci-runner
    spec:
      volumes:
      - name: gitlab-ci-runner-scripts
        projected:
          sources:
          - configMap:
              name: gitlab-ci-runner-scripts
              items:
              - key: run.sh
                path: run.sh
                mode: 0755
      serviceAccountName: gitlab-ci
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        supplementalGroups: [999]
      containers:
      - image: 172.16.60.230/gitlab/gitlab-runner:latest
        name: gitlab-ci-runner
        command:
        - /scripts/run.sh
        envFrom:
        - configMapRef:
            name: gitlab-ci-runner-cm
        - secretRef:
            name: gitlab-ci-token
        env:
        - name: RUNNER_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 9100
          name: http-metrics
          protocol: TCP
        volumeMounts:
        - name: gitlab-ci-runner-scripts
          mountPath: "/scripts"
          readOnly: true
      restartPolicy: Always

上面我们命名了一个gitlab-ci的serviceAccount，这里要新建一个rbac文件 (runner-rbac.yaml)

[root@k8s-master01 gitlab]# cat runner-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: kube-ops
roleRef:
  kind: Role
  name: gitlab-ci
  apiGroup: rbac.authorization.k8s.io

创建gitlab runner

[root@k8s-master01 gitlab]# kubectl apply -f runner-rbac.yaml
serviceaccount/gitlab-ci created
role.rbac.authorization.k8s.io/gitlab-ci created
rolebinding.rbac.authorization.k8s.io/gitlab-ci created

[root@k8s-master01 gitlab]# kubectl apply -f runner-statefulset.yaml
service/gitlab-ci-runner unchanged
statefulset.apps/gitlab-ci-runner created

接下来我们检查我们创建的资源信息：

[root@k8s-master01 gitlab]# kubectl get pod,svc,cm -n kube-ops
NAME                              READY   STATUS    RESTARTS   AGE
pod/gitlab-5b887894d5-ntxzj       1/1     Running   1          150m
pod/gitlab-ci-runner-0            1/1     Running   0          12m
pod/gitlab-ci-runner-1            1/1     Running   0          8m33s
pod/postgresql-57bf98cdf8-7mdh9   1/1     Running   1          150m
pod/redis-56769dc6b6-c4rnq        1/1     Running   0          150m

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                     AGE
service/gitlab             NodePort    10.254.252.242   <none>        80:30004/TCP,22:31455/TCP   150m
service/gitlab-ci-runner   ClusterIP   None             <none>        9100/TCP                    13m
service/postgresql         ClusterIP   10.254.173.69    <none>        5432/TCP                    150m
service/redis              ClusterIP   10.254.11.213    <none>        6379/TCP                    150m

NAME                                 DATA   AGE
configmap/gitlab-ci-runner-cm        19     36m
configmap/gitlab-ci-runner-scripts   1      32m

此时，在登录Gitlab，查看Runner信息，发现就已经将这2个pod节点添加进来了：