Jenkins在kubernetes上的落地实践

公众号: 云原生生态圈

发布于 2020-06-15 14:33:38

7570

发布于 2020-06-15 14:33:38

文章被收录于专栏：云原生生态圈云原生生态圈云原生生态圈

在k8s上运行jenkins之前，我们需要知道几件事情：

可以通过 image修改jenkins镜像的版本
可以通过 nodeSelector选择jenkins pod运行在哪个Node(cpu/mem充足)节点上
我们对Jenkins的运行做了资源限制，如果不够，可以按需修改， limits和 requests中的CPU单位通常是指CPU的千分之一为最小单位，所以1000m就表示1个CPU,200m表示0.2个
我们对jenkins的数据做了持久化配置，通过nfs提供pv,如果你有ceph,glusterfs，可以按需修改
jenkins在k8s上被指定到了 kube-ops命令空间下，如果没有，请提前创建 kubectl create ns kube-ops
使用jenkins最头疼的事就是插件下载不下来，无法正常运行，因此有两个办法就是，第一，你把 基础插件提前拷贝到存储卷上，第二就是通过把插件打到jenkins docker镜像内，其实都一样。

准备配置清单

配置清单的文件名称，使用者按需自己创建。

准备pv和pvc的清单配置文件

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkinspv
spec:
  capacity:
    storage: 15Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 172.16.132.231
    path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkinspvc
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 15G

创建pv/pvc对象,这里我们要注意nfs提供给jenkins的存储目录的权限问题,否则服务因为权限无法写入数据:

在nfs服务主机上:

chown -R 1000 /data/k8s/jenkins2

然后，我们就可以配置pv了：

kubectl apply -f jenkins-storage.yaml

准备ServiceAccount资源对象

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins2
  namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins2
  namespace: kube-ops
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins2
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins2
subjects:
  - kind: ServiceAccount
    name: jenkins2
    namespace: kube-ops

创建sa对象

kubectl apply -f jenkins-sa.yaml

准备jenkins deployment配置清单

在准备好sa和存储之后，我们就可以运行jenkins了，这里我们通过 ClusterIP访问Jenkins pod,因此我们还会准备一个jenkins service资源对象在deployment下方

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins2
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app: jenkins2
  template:
    metadata:
      labels:
        app: jenkins2
    spec:
      nodeSelector:
        kubernetes.io/hostname: dev-k8s-02.kubemaster.top
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins2
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 2000m
            memory: 4Gi
          requests:
            cpu: 1000m
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkinshome
          subPath: jenkins2
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: jenkinspvc

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins2
  namespace: kube-ops
  labels:
    app: jenkins2
spec:
  selector:
    app: jenkins2
  ports:
  - name: web
    port: 8080
    targetPort: web
  - name: agent
    port: 50000
    targetPort: agent

此时我们就可以应用 apply这个deployment了，在应用之后，如果你的 nodeSelector服务器上没有这个jenkins的镜像，可能需要一段时间拉取

kubectl apply -f jenkins-deployment-with-svc.yaml

当jenkins运行成功之后，我们应该如何访问呢，此处我们通过traefik的 ingressroute增加一个路由，用于在(k8s集群外)外部访问jenkins。

准备jenkins ingressroute配置清单

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: jenkinsingress
  namespace: kube-ops
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`jenkins.kubemaster.top`)
      kind: Rule
      services:
        - name: jenkins2
          port: 8080

将配置应用到k8s环境中

kubectl apply -f jenkins-ingressroute.yaml

准备域名以及DNS解析

准备一个域名，将域名解析到Traefik运行的主机上，通过web端口既可以访问jenkins了

☸️  devcluster? tracing  ~/api   master  ? ? curl -I jenkins.kubemaster.top
HTTP/1.1 403 Forbidden
Content-Length: 793
Content-Type: text/html;charset=utf-8
Date: Tue, 21 Apr 2020 09:32:36 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Server: Jetty(9.4.25.v20191220)
Set-Cookie: JSESSIONID.b1161c1c=node07msgdvrmibs59emvt3tn25pp433368.node0; Path=/; HttpOnly
X-Content-Type-Options: nosniff
X-Hudson: 1.395
X-Hudson-Cli-Port: 50000
X-Jenkins: 2.204.3
X-Jenkins-Cli-Port: 50000
X-Jenkins-Cli2-Port: 50000
X-Jenkins-Session: d1c63459
X-Permission-Implied-By: hudson.security.Permission.GenericRead
X-Permission-Implied-By: hudson.model.Hudson.Administer
X-Required-Permission: hudson.model.Hudson.Read
X-You-Are-Authenticated-As: anonymous
X-You-Are-In-Group-Disabled: JENKINS-39402: use -Dhudson.security.AccessDeniedException2.REPORT_GROUP_HEADERS=true or use /whoAmI to diagnose