专栏首页JadePeng的技术博客drone的pipeline原理与代码分析

drone的pipeline原理与代码分析

最近的一个项目,需要实现一个工作任务流(task pipeline),基于之前CICD的经验,jenkins pipeline和drone的pipeline进入候选。

drone是基于go的cicd解决方案,github上有1.6万+star,本文简单对比了其和jenkins的区别,重点介绍了drone的pipeline原理,并简单分析了代码。

jenkins 与 drone

对比项

jenkins

drone

pipeline定义

编写jenkinsfile

编写流程yml

运行方式

在一个pod里运行

每一步骤起对应的container,通过挂载volume实现数据共享

运行环境

物理机或者容器环境,包括K8S

docker容器环境

开发语言

java

golang

drone pipeline好处是相对更轻量级,yml定义也相对简洁清晰,按照功能来划分容器,可以方便的实现task的复用,而jenkins则是完全打包到一个镜像,会造成单个镜像体积过大,比如jenkins的单个镜像超过2G。

drone的pipeline,是基于https://github.com/cncd/pipeline 实现的,这里简单分析下其原理。

编译和执行 drone pipeline

要了解一个程序的原理,先从输入输出讲起。

先安装:

go get -u github.com/cncd/pipeline
go install github.com/cncd/pipeline/pipec

然后测试

cd $GOPATH/github.com/cncd/pipeline/samples/sample_1
# ll
total 28
drwxr-xr-x  2 root root 4096 Jan 22 11:44 ./
drwxr-xr-x 13 root root 4096 Jan 22 11:02 ../
-rw-r--r--  1 root root  549 Jan 22 11:02 .env
-rw-r--r--  1 root root 6804 Jan 22 16:30 pipeline.json
-rw-r--r--  1 root root  229 Jan 22 11:02 pipeline.yml
-rw-r--r--  1 root root  138 Jan 22 11:02 README.md
  • pipeline.yml 定义文件
  • pipeline.json 编译后的配置文件
  • .env 环境变量

先来查看pipeline.yml 定义

workspace:
  base: /go
  path: src/github.com/drone/envsubst

clone:
  git:
    image: plugins/git
    depth: 50

pipeline:
  build:
    image: golang:1.7
    commands:
      - go get -t ./...
      - go build
      - go test -v

上面的yml定义了:

  • 工作目录workspace
  • 初始化工作,git clone仓库,仓库地址在.env里定义
  • 然后是定义pipeline,
    • pipeline下面是step数组,这里只有一个build
    • 使用golang:1.7镜像
    • 构建命令在commands数组里定义

通过pipec compilecompile配置文件:

# pipec compile
Successfully compiled pipeline.yml to pipeline.json

查看编译后的pipeline.json

{
  "pipeline": [
    {
      "name": "pipeline_clone_0",
      "alias": "git",
      "steps": [
        {
          "name": "pipeline_clone_0",
          "alias": "git",
          "image": "plugins/git:latest",
          "working_dir": "/go/src/github.com/drone/envsubst",
          "environment": {
            "CI": "drone",
            "CI_BUILD_CREATED": "1486119586",
            "CI_BUILD_EVENT": "push",
            "CI_BUILD_NUMBER": "6",
            "CI_BUILD_STARTED": "1486119585",
            "CI_COMMIT_AUTHOR": "bradrydzewski",
            "CI_COMMIT_AUTHOR_NAME": "bradrydzewski",
            "CI_COMMIT_BRANCH": "master",
            "CI_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "CI_COMMIT_REF": "refs/heads/master",
            "CI_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "CI_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "CI_REPO": "drone/envsubst",
            "CI_REPO_LINK": "https://github.com/drone/envsubst",
            "CI_REPO_NAME": "drone/envsubst",
            "CI_REPO_REMOTE": "https://github.com/drone/envsubst.git",
            "CI_SYSTEM": "pipec",
            "CI_SYSTEM_ARCH": "linux/amd64",
            "CI_SYSTEM_LINK": "https://github.com/cncd/pipec",
            "CI_SYSTEM_NAME": "pipec",
            "CI_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "DRONE": "true",
            "DRONE_ARCH": "linux/amd64",
            "DRONE_BRANCH": "master",
            "DRONE_BUILD_CREATED": "1486119586",
            "DRONE_BUILD_EVENT": "push",
            "DRONE_BUILD_LINK": "https://github.com/cncd/pipec/drone/envsubst/6",
            "DRONE_BUILD_NUMBER": "6",
            "DRONE_BUILD_STARTED": "1486119585",
            "DRONE_COMMIT": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_COMMIT_AUTHOR": "bradrydzewski",
            "DRONE_COMMIT_BRANCH": "master",
            "DRONE_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "DRONE_COMMIT_REF": "refs/heads/master",
            "DRONE_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_JOB_STARTED": "1486119585",
            "DRONE_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "DRONE_REPO": "drone/envsubst",
            "DRONE_REPO_LINK": "https://github.com/drone/envsubst",
            "DRONE_REPO_NAME": "envsubst",
            "DRONE_REPO_OWNER": "drone",
            "DRONE_REPO_SCM": "git",
            "DRONE_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "PLUGIN_DEPTH": "50"
          },
          "volumes": [
            "pipeline_default:/go"
          ],
          "networks": [
            {
              "name": "pipeline_default",
              "aliases": [
                "git"
              ]
            }
          ],
          "on_success": true,
          "auth_config": {}
        }
      ]
    },
    {
      "name": "pipeline_stage_0",
      "alias": "build",
      "steps": [
        {
          "name": "pipeline_step_0",
          "alias": "build",
          "image": "golang:1.7",
          "working_dir": "/go/src/github.com/drone/envsubst",
          "environment": {
            "CI": "drone",
            "CI_BUILD_CREATED": "1486119586",
            "CI_BUILD_EVENT": "push",
            "CI_BUILD_NUMBER": "6",
            "CI_BUILD_STARTED": "1486119585",
            "CI_COMMIT_AUTHOR": "bradrydzewski",
            "CI_COMMIT_AUTHOR_NAME": "bradrydzewski",
            "CI_COMMIT_BRANCH": "master",
            "CI_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "CI_COMMIT_REF": "refs/heads/master",
            "CI_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "CI_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "CI_REPO": "drone/envsubst",
            "CI_REPO_LINK": "https://github.com/drone/envsubst",
            "CI_REPO_NAME": "drone/envsubst",
            "CI_REPO_REMOTE": "https://github.com/drone/envsubst.git",
            "CI_SCRIPT": "CmlmIFsgLW4gIiRDSV9ORVRSQ19NQUNISU5FIiBdOyB0aGVuCmNhdCA8PEVPRiA+ICRIT01FLy5uZXRyYwptYWNoaW5lICRDSV9ORVRSQ19NQUNISU5FCmxvZ2luICRDSV9ORVRSQ19VU0VSTkFNRQpwYXNzd29yZCAkQ0lfTkVUUkNfUEFTU1dPUkQKRU9GCmNobW9kIDA2MDAgJEhPTUUvLm5ldHJjCmZpCnVuc2V0IENJX05FVFJDX1VTRVJOQU1FCnVuc2V0IENJX05FVFJDX1BBU1NXT1JECnVuc2V0IENJX1NDUklQVAp1bnNldCBEUk9ORV9ORVRSQ19VU0VSTkFNRQp1bnNldCBEUk9ORV9ORVRSQ19QQVNTV09SRAoKZWNobyArICJnbyBnZXQgLXQgLi8uLi4iCmdvIGdldCAtdCAuLy4uLgoKZWNobyArICJnbyBidWlsZCIKZ28gYnVpbGQKCmVjaG8gKyAiZ28gdGVzdCAtdiIKZ28gdGVzdCAtdgoK",
            "CI_SYSTEM": "pipec",
            "CI_SYSTEM_ARCH": "linux/amd64",
            "CI_SYSTEM_LINK": "https://github.com/cncd/pipec",
            "CI_SYSTEM_NAME": "pipec",
            "CI_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "DRONE": "true",
            "DRONE_ARCH": "linux/amd64",
            "DRONE_BRANCH": "master",
            "DRONE_BUILD_CREATED": "1486119586",
            "DRONE_BUILD_EVENT": "push",
            "DRONE_BUILD_LINK": "https://github.com/cncd/pipec/drone/envsubst/6",
            "DRONE_BUILD_NUMBER": "6",
            "DRONE_BUILD_STARTED": "1486119585",
            "DRONE_COMMIT": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_COMMIT_AUTHOR": "bradrydzewski",
            "DRONE_COMMIT_BRANCH": "master",
            "DRONE_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "DRONE_COMMIT_REF": "refs/heads/master",
            "DRONE_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_JOB_STARTED": "1486119585",
            "DRONE_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "DRONE_REPO": "drone/envsubst",
            "DRONE_REPO_LINK": "https://github.com/drone/envsubst",
            "DRONE_REPO_NAME": "envsubst",
            "DRONE_REPO_OWNER": "drone",
            "DRONE_REPO_SCM": "git",
            "DRONE_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "HOME": "/root",
            "SHELL": "/bin/sh"
          },
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "echo $CI_SCRIPT | base64 -d | /bin/sh -e"
          ],
          "volumes": [
            "pipeline_default:/go"
          ],
          "networks": [
            {
              "name": "pipeline_default",
              "aliases": [
                "build"
              ]
            }
          ],
          "on_success": true,
          "auth_config": {}
        }
      ]
    }
  ],
  "networks": [
    {
      "name": "pipeline_default",
      "driver": "bridge"
    }
  ],
  "volumes": [
    {
      "name": "pipeline_default",
      "driver": "local"
    }
  ],
  "secrets": null
}

简单分析结构:

  • pipeline 定义了执行的stage,每个stage有一个或者多个step
  • networks、volumes、secrets 分别定义网络、存储和secrets
    • 通过network,实现container互通
    • 通过volumes实现数据共享

最后执行,通过pipec exec

# pipec exec
proc "pipeline_clone_0" started
+ git init
Initialized empty Git repository in /go/src/github.com/drone/envsubst/.git/
+ git remote add origin https://github.com/drone/envsubst.git
+ git fetch --no-tags --depth=50 origin +refs/heads/master:
From https://github.com/drone/envsubst
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
+ git reset --hard -q d0876d3176965f9552a611cbd56e24a9264355e6
+ git submodule update --init --recursive
proc "pipeline_clone_0" exited with status 0
proc "pipeline_step_0" started
+ go get -t ./...
+ go build
+ go test -v
=== RUN   TestExpand
--- PASS: TestExpand (0.00s)
=== RUN   TestFuzz
--- PASS: TestFuzz (0.01s)
=== RUN   Test_len
--- PASS: Test_len (0.00s)
=== RUN   Test_lower
--- PASS: Test_lower (0.00s)
=== RUN   Test_lowerFirst
--- PASS: Test_lowerFirst (0.00s)
=== RUN   Test_upper
--- PASS: Test_upper (0.00s)
=== RUN   Test_upperFirst
--- PASS: Test_upperFirst (0.00s)
=== RUN   Test_default
--- PASS: Test_default (0.00s)
PASS
ok      github.com/drone/envsubst   0.009s
proc "pipeline_step_0" exited with status 0

pipeline 原理分析

编译过程

可以形象的理解为 .env+pipeline.yml --> pipeline.json

编译过程不复杂,主要是解析pipeline.yml为Config:

Config struct {
        Cache     libcompose.Stringorslice
        Platform  string
        Branches  Constraint
        Workspace Workspace
        Clone     Containers
        Pipeline  Containers
        Services  Containers
        Networks  Networks
        Volumes   Volumes
        Labels    libcompose.SliceorMap
    }

然后转换为json对应的config:

Config struct {
        Stages   []*Stage   `json:"pipeline"` // pipeline stages
        Networks []*Network `json:"networks"` // network definitions
        Volumes  []*Volume  `json:"volumes"`  // volume definitions
        Secrets  []*Secret  `json:"secrets"`  // secret definitions
    }

该部分主要代码在pipeline/frontend里

执行过程

我们主要关注执行过程,主要代码在pipeline/backend里。

首先是读取配置文件为backend.Config

config, err := pipeline.Parse(reader)
    if err != nil {
        return err
    }

然后创建执行环境,目前的代码仅docker可用,k8s是空代码。

var engine backend.Engine
    if c.Bool("kubernetes") {
        engine = kubernetes.New(
            c.String("kubernetes-namepsace"),
            c.String("kubernetes-endpoint"),
            c.String("kubernetes-token"),
        )
    } else {
        engine, err = docker.NewEnv()
        if err != nil {
            return err
        }
    }

接着开始执行

    ctx, cancel := context.WithTimeout(context.Background(), c.Duration("timeout"))
    defer cancel()
    ctx = interrupt.WithContext(ctx)

    return pipeline.New(config,
        pipeline.WithContext(ctx),
        pipeline.WithLogger(defaultLogger),
        pipeline.WithTracer(defaultTracer),
        pipeline.WithEngine(engine),
    ).Run()

其中pipeline.NEW创建了Runtime对象;

type Runtime struct {
    err     error  // 错误信息
    spec    *backend.Config  // 配置信息
    engine  backend.Engine  // docker engine
    started int64 // 开始时间

    ctx    context.Context
    tracer Tracer
    logger Logger
}

其中Engine,操作容器的interface,目前仅docker可用。

// Engine defines a container orchestration backend and is used
// to create and manage container resources.
type Engine interface {
    // Setup the pipeline environment.
    Setup(context.Context, *Config) error
    // Start the pipeline step.
    Exec(context.Context, *Step) error
    // Kill the pipeline step.
    Kill(context.Context, *Step) error
    // Wait for the pipeline step to complete and returns
    // the completion results.
    Wait(context.Context, *Step) (*State, error)
    // Tail the pipeline step logs.
    Tail(context.Context, *Step) (io.ReadCloser, error)
    // Destroy the pipeline environment.
    Destroy(context.Context, *Config) error
}

关注Run:

// Run starts the runtime and waits for it to complete.
func (r *Runtime) Run() error {
    
    // 延迟函数,用于销毁docker env
    defer func() {
        r.engine.Destroy(r.ctx, r.spec)
    }()
    
    // 初始化docker engine
    r.started = time.Now().Unix()
    if err := r.engine.Setup(r.ctx, r.spec); err != nil {
        return err
    }
   
   // 依次运行stage
    for _, stage := range r.spec.Stages {
        select {
        case <-r.ctx.Done():
            return ErrCancel
        // 执行
        case err := <-r.execAll(stage.Steps):
            if err != nil {
                r.err = err
            }
        }
    }

    return r.err
}

重点在于使用errgroup.Group通过协程方式运行step:

// 执行所有steps
func (r *Runtime) execAll(procs []*backend.Step) <-chan error {
    var g errgroup.Group
    done := make(chan error)
    // 遍历执行step
    for _, proc := range procs {
       // 协程 exec
        proc := proc
        g.Go(func() error {
            return r.exec(proc)
        })
    }

    go func() {
        done <- g.Wait()
        close(done)
    }()
    return done
}

// 执行单个step
func (r *Runtime) exec(proc *backend.Step) error {
    switch {
    case r.err != nil && proc.OnFailure == false:
        return nil
    case r.err == nil && proc.OnSuccess == false:
        return nil
    }
    
    // trace日志
    if r.tracer != nil {
        state := new(State)
        state.Pipeline.Time = r.started
        state.Pipeline.Error = r.err
        state.Pipeline.Step = proc
        state.Process = new(backend.State) // empty
        if err := r.tracer.Trace(state); err == ErrSkip {
            return nil
        } else if err != nil {
            return err
        }
    }
   
   // docker engine执行
    if err := r.engine.Exec(r.ctx, proc); err != nil {
        return err
    }
   
   // 记录日志信息
    if r.logger != nil {
        rc, err := r.engine.Tail(r.ctx, proc)
        if err != nil {
            return err
        }

        go func() {
            r.logger.Log(proc, multipart.New(rc))
            rc.Close()
        }()
    }

    if proc.Detached {
        return nil
    }

   // 等待docker engine执行完成
    wait, err := r.engine.Wait(r.ctx, proc)
    if err != nil {
        return err
    }

    if r.tracer != nil {
        state := new(State)
        state.Pipeline.Time = r.started
        state.Pipeline.Error = r.err
        state.Pipeline.Step = proc
        state.Process = wait
        if err := r.tracer.Trace(state); err != nil {
            return err
        }
    }
   
    if wait.OOMKilled {
        return &OomError{
            Name: proc.Name,
            Code: wait.ExitCode,
        }
    } else if wait.ExitCode != 0 {
        return &ExitError{
            Name: proc.Name,
            Code: wait.ExitCode,
        }
    }
    return nil
}

作者:Jadepeng 出处:jqpeng的技术记事本--http://www.cnblogs.com/xiaoqi 您的支持是对博主最大的鼓励,感谢您的认真阅读。 本文版权归作者所有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,否则保留追究法律责任的权利。

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • Spring Security 架构与源码分析

    Spring Security 主要实现了Authentication(认证,解决who are you? ) 和 Access Control(访问控制,也就...

    用户1177380
  • 正则匹配拼音

    把可能的列出来就可以了,因此正则表达式这样写: [āáǎàēéěèīíǐìōóǒòūúǔùǖǘǚǜüêɑńňɡa-zA-ZA-Za-z\\s∥-]+ 在c#中...

    用户1177380
  • Docker启用TLS进行安全配置

    之前开启了docker的2375 Remote API,接到公司安全部门的要求,需要启用授权,翻了下官方文档

    用户1177380
  • shell入门系列( 二 )别名

    版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。

    suveng
  • golang与redis的基本使用

    上面是golang 连接redis 的基本代码。连接redis 数据库 方式跟net包连接网络方式样子很想。 认证:如果redis 有密码 ,在连接成功后必...

    地球流浪猫
  • Coinbase大变!它要与‘’垃圾代币‘’同流合污了?

    受美国证券交易委员会(SEC)的监管影响,Coinbase 是加密数字货币领域最保守的交易所之一,这家市值80亿美元的公司仅仅向用户提供了不超过10种的数字加密...

    区块链大本营
  • SpringBoot系列教程JPA之基础环境搭建

    JPA(Java Persistence API)Java持久化API,是 Java 持久化的标准规范,Hibernate是持久化规范的技术实现,而Spring...

    一灰灰blog
  • 数据的艺术 Teradata数据科学家数据可视化作品集

    近日,在Teradata大数据峰会上展出了由Teradata数据科学家及数据顾问提供的一系列的”数据分析艺术”数据分析视觉化展,继阿姆斯特丹Teradata U...

    CSDN技术头条
  • 001.YUM源服务端搭建

    提示:本环境方便测试,将服务器地址添加至hosts,生产环境在有DNS情况下可跳过。

    木二
  • IdentityServer4实战 - JWT Token Issuer 详解

    众所周知 JWT Token 由三部分组成,第一部分 Header,包含 keyid、签名算法、Token类型;第二部分 Payload 包含 Token 的信...

    晓晨

扫码关注云+社区

领取腾讯云代金券