【GitLab CI/CD】：Pipelines

WEBJ2EE

发布于 2021-01-18 14:44:02

7790

发布于 2021-01-18 14:44:02

文章被收录于专栏：WebJ2EE

目录
1. Pipelines 概述
2. Pipelines 常用类型
3. Pipelines 常用配置
    3.1. Git strategy
    3.2. Maximum artifacts size
4. Pipelines 架构
    4.1. Basic Pipelines
    4.2. Directed Acyclic Graph Pipelines
    4.3. Child / Parent Pipelines
5. 综合示例

1. Pipelines 概述

Pipelines are the top-level component of continuous integration, delivery, and deployment.

Pipelines comprise:

Jobs, which define what to do. For example, jobs that compile or test code.
Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code.

Jobs are executed by runners.Multiple jobs in the same stage are executed in parallel, if there are enough concurrent runners.

If all jobs in a stage succeed, the pipeline moves on to the next stage.

If any job in a stage fails, the next stage is not (usually) executed and the pipeline ends early.

In general, pipelines are executed automatically and require no intervention once created. However, there are also times when you can manually interact with a pipeline.

A typical pipeline might consist of four stages, executed in the following order:

A build stage, with a job called compile.
A test stage, with two jobs called test1 and test2.
A staging stage, with a job called deploy-to-stage.
A production stage, with a job called deploy-to-prod.

2. Pipelines 常用类型

Pipelines can be configured in many different ways:

Basic pipelines run everything in each stage concurrently, followed by the next stage.
Directed Acyclic Graph Pipeline (DAG) pipelines are based on relationships between jobs and can run more quickly than basic pipelines.
Multi-project pipelines combine pipelines for different projects together.
Parent-Child pipelines break down complex pipelines into one parent pipeline that can trigger multiple child sub-pipelines, which all run in the same project and with the same SHA.

3. Pipelines 常用配置

3.1. Git strategy

With Git strategy, you can choose the default way your repository is fetched from GitLab in a job.

There are two options. Using:

git clone, which is slower since it clones the repository from scratch for every job, ensuring that the local working copy is always pristine.
git fetch, which is default in GitLab and faster as it re-uses the local working copy (falling back to clone if it doesn’t exist). This is recommended, especially for large repositories.

The configured Git strategy can be overridden by the GIT_STRATEGY variable in .gitlab-ci.yml.

3.2. Maximum artifacts size

4. Pipelines 架构

4.1. Basic Pipelines

This is the simplest pipeline in GitLab. It runs everything in the build stage concurrently, and once all of those finish, it runs everything in the test stage the same way, and so on. It’s not the most efficient, and if you have lots of steps it can grow quite complex, but it’s easier to maintain:

stages:
  - build
  - test
  - deploy

image: alpine

build_a:
  stage: build
  script:
    - echo "This job builds something."

build_b:
  stage: build
  script:
    - echo "This job builds something else."

test_a:
  stage: test
  script:
    - echo "This job tests something. It will only run when all jobs in the"
    - echo "build stage are complete."

test_b:
  stage: test
  script:
    - echo "This job tests something else. It will only run when all jobs in the"
    - echo "build stage are complete too. It will start at about the same time as test_a."

deploy_a:
  stage: deploy
  script:
    - echo "This job deploys something. It will only run when all jobs in the"
    - echo "test stage complete."

deploy_b:
  stage: deploy
  script:
    - echo "This job deploys something else. It will only run when all jobs in the"
    - echo "test stage complete. It will start at about the same time as deploy_a."

4.2. Directed Acyclic Graph Pipelines

If efficiency is important to you and you want everything to run as quickly as possible, you can use Directed Acyclic Graphs (DAG). Use the needs keyword to define dependency relationships between your jobs.When GitLab knows the relationships between your jobs, it can run everything as fast as possible, and even skips into subsequent stages when possible.

In the example below, if build_a and test_a are much faster than build_b and test_b, GitLab starts deploy_a even if build_b is still running.

stages:
  - build
  - test
  - deploy

image: alpine

build_a:
  stage: build
  script:
    - echo "This job builds something quickly."

build_b:
  stage: build
  script:
    - echo "This job builds something else slowly."

test_a:
  stage: test
  needs: [build_a]
  script:
    - echo "This test job will start as soon as build_a finishes."
    - echo "It will not wait for build_b, or other jobs in the build stage, to finish."

test_b:
  stage: test
  needs: [build_b]
  script:
    - echo "This test job will start as soon as build_b finishes."
    - echo "It will not wait for other jobs in the build stage to finish."

deploy_a:
  stage: deploy
  needs: [test_a]
  script:
    - echo "Since build_a and test_a run quickly, this deploy job can run much earlier."
    - echo "It does not need to wait for build_b or test_b."

deploy_b:
  stage: deploy
  needs: [test_b]
  script:
    - echo "Since build_b and test_b run slowly, this deploy job will run much later."

4.3. Child / Parent Pipelines

In the examples above, it’s clear we’ve got two types of things that could be built independently. This is an ideal case for using Child / Parent Pipelines) via the trigger keyword. It separates out the configuration into multiple files, keeping things very simple. You can also combine this with:

The rules keyword: For example, have the child pipelines triggered only when there are changes to that area.
The include keyword: Bring in common behaviors, ensuring you are not repeating yourself.
DAG pipelines inside of child pipelines, achieving the benefits of both.

Example /.gitlab-ci.yml configuration for the parent pipeline matching the diagram:

stages:
  - triggers

trigger_a:
  stage: triggers
  trigger:
    include: a/.gitlab-ci.yml
  rules:
    - changes:
        - a/*

trigger_b:
  stage: triggers
  trigger:
    include: b/.gitlab-ci.yml
  rules:
    - changes:
        - b/*

Example child a pipeline configuration, located in /a/.gitlab-ci.yml, making use of the DAG needs: keyword:

stages:
  - build
  - test
  - deploy

image: alpine

build_a:
  stage: build
  script:
    - echo "This job builds something."

test_a:
  stage: test
  needs: [build_a]
  script:
    - echo "This job tests something."

deploy_a:
  stage: deploy
  needs: [test_a]
  script:
    - echo "This job deploys something."

Example child b pipeline configuration, located in /b/.gitlab-ci.yml, making use of the DAG needs: keyword:

stages:
  - build
  - test
  - deploy

image: alpine

build_b:
  stage: build
  script:
    - echo "This job builds something else."

test_b:
  stage: test
  needs: [build_b]
  script:
    - echo "This job tests something else."

deploy_b:
  stage: deploy
  needs: [test_b]
  script:
    - echo "This job deploys something else."

It’s also possible to set jobs to run before or after triggering child pipelines, for example if you have common setup steps or a unified deployment at the end.

5. 综合示例

.gitlab-ci.yml：

stages:
  - deploy

target-1:
  stage: deploy
  trigger:
    include: .gitlab-ci_deploy.yml
  variables:
    DEPLOY_SERVER: "root@a.b.c.d"
    TOMCAT_PATH: "/home/tomcat/tomcat8-7300-sep"
    WAR_NAME: "SEP.war"
  rules:
    - if: '$CI_COMMIT_MESSAGE =~ /^deploy/'

target-2:
  stage: deploy
  trigger:
    include: .gitlab-ci_deploy.yml
  variables:
    DEPLOY_SERVER: "root@o.p.q.r"
    TOMCAT_PATH: "/home/apache-tomcat-8.0.52-7300"
    WAR_NAME: "SEP.war"
  rules:
    - if: '$CI_COMMIT_MESSAGE =~ /^deploy/'

‍‍.gitlab-ci_deploy.yml：

stages:
  - build
  - deploy

build-sep:
  stage: build
  script:
    - mvn -q -Dmaven.test.skip=true clean deploy
  artifacts:
    paths:
      - target/SEP.war

deploy-sep:
  stage: deploy
  script:
    ## 查看环境
    - echo "${DEPLOY_SERVER}; ${TOMCAT_PATH}; ${WAR_NAME}; ${POLARIS_URL}"
    ## 停止Tomcat
    - ssh ${DEPLOY_SERVER} "export kpid=\`ps -ef| grep ${TOMCAT_PATH} | grep -v grep | awk '{print \$2}'\`;for i in \$kpid; do kill -9 \$i;done"
    - ssh ${DEPLOY_SERVER} "sleep 5"
    ## 备份war包
    - ssh ${DEPLOY_SERVER} "mkdir ${TOMCAT_PATH}/back-${CI_COMMIT_REF_NAME}-${CI_JOB_ID}"
    - ssh ${DEPLOY_SERVER} "if [ -f "${TOMCAT_PATH}/webapps/${WAR_NAME}" ];then mv ${TOMCAT_PATH}/webapps/${WAR_NAME} ${TOMCAT_PATH}/back-${CI_COMMIT_REF_NAME}-${CI_JOB_ID};fi"
    ## 清理Tomcat缓存
    - ssh ${DEPLOY_SERVER} "rm -rf ${TOMCAT_PATH}/temp/*"
    - ssh ${DEPLOY_SERVER} "rm -rf ${TOMCAT_PATH}/work/*"
    - ssh ${DEPLOY_SERVER} "rm -rf ${TOMCAT_PATH}/logs/*"
    - ssh ${DEPLOY_SERVER} "rm -rf ${TOMCAT_PATH}/bin/logs/*"
    - ssh ${DEPLOY_SERVER} "rm -rf ${TOMCAT_PATH}/webapps/SEP.war"
    - ssh ${DEPLOY_SERVER} "rm -rf ${TOMCAT_PATH}/webapps/SEP"
    ## 部署WAR包
    - scp target/${WAR_NAME} ${DEPLOY_SERVER}:${TOMCAT_PATH}/webapps
    ## 传送环境变量
    - ssh ${DEPLOY_SERVER} "echo 'export JAVA_OPTS=\"-Xms1024m -Xmx1024m -Dfile.encoding=UTF-8\"' > ${TOMCAT_PATH}/bin/setenv.sh"
    ## 启动Tomcat
    - ssh ${DEPLOY_SERVER} "source /etc/profile; sh ${TOMCAT_PATH}/bin/startup.sh"