Gitlab CI/CD 实践一：Gitlab Runner 安装到 K8S 集群

Yuyy

发布于 2022-09-21 10:26:14

3K0

发布于 2022-09-21 10:26:14

文章被收录于专栏：yuyy.info技术专栏

前言

Gitlab Runner可以直接使用二进制、Docker或者k8s来部署，而使用k8s部署带来的的好处是：合理利用资源，工作容器会被调度到资源相对空闲的节点（构建是一个比较耗费资源的过程）。

创建单独的namespace

gitlab-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: gitlab

注册Runner

Gitlab Runner有3种级别

全局共享
1. 因为executor使用的是容器，不是shell，所以非特殊要求，使用这个就行。如果是shell，可能每个项目用到的环境不同，需要单独使用runner
群组共享
项目独占

Runner的并发性

每个job会单独起一个容器
不同流水线的job是并行处理
同一流水线同一阶段的job也是并行处理

获取 Gitlab CI Register Token

访问全局Runner配置地址：https://xxx.com/admin/runners

gitlab-ci-token-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: gitlab-ci-token
  namespace: gitlab
  labels:
    app: gitlab-ci-runner
data:
  GITLAB_CI_TOKEN: XXXGYTdVdE1zc2lXeXpXcXRVTXI=

GITLAB_CI_TOKEN：Gitlab CI Register Token的base64编码

配置存储

创建对象存储bucket

因构建过程需要缓存一些文件，例如依赖，所以要给Runner配置存储。这里用到的分布式存储是ceph，首先创建Runner专用的账户和对象存储bucket，参照这篇教程：Go项目基于Gitlab CI/CD实践二：Rook Ceph创建S3 bucket用于Gitlab Runner缓存。

gitlab-runner-cache-s3-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: gitlab-runner-cache-s3
  namespace: gitlab
  labels:
    app: gitlab-ci-runner
data:
  CACHE_S3_ACCESS_KEY: "MDZTSEs5T05OUkxYQjdLU1E0UDM="
  CACHE_S3_SECRET_KEY: "QXVkc3JhQlN3alZtNWhqTFo3WG9CbUE2UGU1Q2o3SkJZblQ0R3lUQw=="

CACHE_S3_ACCESS_KEY：对象存储bucket的AKbase64编码
CACHE_S3_SECRET_KEY：对象存储bucket的SKbase64编码

配置SSL证书

由于gitlab服务是https，runner访问gitlab的注册接口时，需要证书。

获取SSL证书

如果gitlab是docker部署的，ssl证书所在路径为：/etc/gitlab/ssl/xxx.com.crt

gitlab-certs-configmap.yaml

apiVersion: v1
data:
  xxx.com.crt: |
    -----BEGIN CERTIFICATE-----
    MIIFSjCCBDKgAwIBAgIUSmsUy8IIhVT1Vl+RbRuO2DY5F3QwDQYJKoZIhvcNAQEN...
    +xxxnphZ/4JE3n3OKiw=
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-certs
  namespace: gitlab

配置注册&注销脚本

默认只有当 Pod 正常通过 Kubernetes（TERM信号）终止时，才会触发Runner取消注册。如果强制终止 Pod（SIGKILL信号），Runner 将不会注销自身。必须手动清理这种被杀死的 Runner 。

gitlab-runner-scripts-configmap.yaml

apiVersion: v1
data:
  run.sh: |
    #!/bin/bash
    unregister() {
        kill %1
        echo "Unregistering runner ${RUNNER_NAME} ..."
        /usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
        exit $?
    }
    trap 'unregister' EXIT HUP INT QUIT PIPE TERM
    mkdir -p /home/gitlab-runner/.gitlab-runner/certs
    cp /certs/xxx.com.crt /home/gitlab-runner/.gitlab-runner/certs/
    echo "Registering runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
    sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /home/gitlab-runner/.gitlab-runner/config.toml

    cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF
            [[runners.kubernetes.volumes.host_path]]
              name = "docker"
              mount_path = "/var/run/docker.sock"
              read_only = true
              host_path = "/var/run/docker.sock"
    EOF

    echo "Starting runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
    wait
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-scripts
  namespace: gitlab

遇到的坑

我把证书的configmap挂载到/certs/（这个步骤后面会提到），然后在上面的启动脚本里，将证书文件从/certs/拷贝到/home/gitlab-runner/.gitlab-runner/certs/，那为什么不直接把证书挂载到/home/gitlab-runner/.gitlab-runner/certs/？

如果这么干，/home/gitlab-runner/.gitlab-runner/certs目录的所有者就是root，而runner容器没有用root用户运行，用的是gitlab-runner用户。

在执行注册脚本/usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}成功时，会创建runner的配置文件/home/gitlab-runner/.gitlab-runner/config.toml。这时就会报错：PANIC: open /home/gitlab-runner/.gitlab-runner/config.toml: permission denied，原因就是gitlab-runner用户操作root创建的目录造成的权限问题。

配置RBAC

gitlab-runner-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: gitlab
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: gitlab
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: gitlab
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: gitlab
roleRef:
  kind: Role
  name: gitlab-ci
  apiGroup: rbac.authorization.k8s.io

配置Runner

gitlab-runner-configmap.yaml

apiVersion: v1
data:
  CACHE_TYPE: "s3"
  CACHE_SHARED: "true"
  CACHE_S3_SERVER_ADDRESS: "rook-ceph-rgw-my-store.rook-ceph.svc"
  CACHE_S3_BUCKET_NAME: "gitlab-runner-cache-bucket"
  CACHE_S3_INSECURE: "true"
  REGISTER_NON_INTERACTIVE: "true"
  REGISTER_LOCKED: "false"
  METRICS_SERVER: "0.0.0.0:9100"
  CI_SERVER_URL: "https://xxx.com/ci"
  RUNNER_REQUEST_CONCURRENCY: "4"
  RUNNER_EXECUTOR: "kubernetes"
  KUBERNETES_NAMESPACE: "gitlab"
  KUBERNETES_PRIVILEGED: "true"
  KUBERNETES_CPU_LIMIT: "3"
  KUBERNETES_MEMORY_LIMIT: "4Gi"
  KUBERNETES_SERVICE_CPU_LIMIT: "3"
  KUBERNETES_SERVICE_MEMORY_LIMIT: "4Gi"
  KUBERNETES_HELPER_CPU_LIMIT: "500m"
  KUBERNETES_HELPER_MEMORY_LIMIT: "500Mi"
  KUBERNETES_PULL_POLICY: "if-not-present"
  KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
  KUBERNETES_POLL_INTERVAL: "5"
  KUBERNETES_POLL_TIMEOUT: "360"
  RUNNER_TAG_LIST: "k8s,100.30.30.192,share"
  RUNNER_NAME: "192-k8s-runner"
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner
  namespace: gitlab

CACHE_S3_SERVER_ADDRESS：ceph对象存储service
CI_SERVER_URL：gitlab地址，如果是在k8s里，可以配置service。记得末尾加上/ci

Runner资源

gitlab-runner-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gitlab-ci-runner
  namespace: gitlab
  labels:
    app: gitlab-ci-runner
spec:
  selector:
    matchLabels:
      app: gitlab-ci-runner      
  updateStrategy:
    type: RollingUpdate
  replicas: 2
  serviceName: gitlab-ci-runner
  template:
    metadata:
      labels:
        app: gitlab-ci-runner
    spec:
      volumes:
      - name: gitlab-ci-runner-scripts
        projected:
          sources:
          - configMap:
              name: gitlab-ci-runner-scripts
              items:
              - key: run.sh
                path: run.sh
                mode: 0755
      - name: gitlab-ci-runner-certs
        projected:
          sources:
          - configMap:
              name: gitlab-ci-runner-certs
              items:
              - key: xxx.com.crt
                path: xxx.com.crt
                mode: 0777
      serviceAccountName: gitlab-ci
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        supplementalGroups: [999]
      containers:
      - image: gitlab/gitlab-runner:latest
        name: gitlab-ci-runner
        command:
        - /scripts/run.sh
        envFrom:
        - configMapRef:
            name: gitlab-ci-runner
        - secretRef:
            name: gitlab-ci-token
        - secretRef:
            name: gitlab-runner-cache-s3
        ports:
        - containerPort: 9100
          name: http-metrics
          protocol: TCP
        volumeMounts:
        - name: gitlab-ci-runner-scripts
          mountPath: "/scripts"
          readOnly: true
        - name: gitlab-ci-runner-certs
          mountPath: "/certs"
      restartPolicy: Always

解决资源依赖关系

kubectl apply是按照资源定义文件的文件名来创建资源的，并不会处理依赖关系。也就是可能出现namespace还没创建，就开始创建其他资源的情况。这里使用Kustomize来保证依赖关系（从 1.14 版本开始，kubectl 也开始支持使用 kustomization 文件来管理 Kubernetes 对象）。

kustomization.yaml

resources:
- gitlab-namespace.yaml
- gitlab-certs-configmap.yaml
- gitlab-ci-token-secret.yaml
- gitlab-runner-cache-s3-secret.yaml
- gitlab-runner-configmap.yaml
- gitlab-runner-rbac.yaml
- gitlab-runner-scripts-configmap.yaml
- gitlab-runner-statefulset.yaml

这里的先后顺序不重要，Kustomize会自动处理

部署

执行命令

进入资源文件目录

.
├── gitlab-certs-configmap.yaml
├── gitlab-ci-token-secret.yaml
├── gitlab-namespace.yaml
├── gitlab-runner-cache-s3-secret.yaml
├── gitlab-runner-configmap.yaml
├── gitlab-runner-rbac.yaml
├── gitlab-runner-scripts-configmap.yaml
├── gitlab-runner-statefulset.yaml
└── kustomization.yaml

执行命令即可：kubectl apply -k .