服务端持续集成实战

用户5521279

发布于 2020-04-01 10:40:32

7490

发布于 2020-04-01 10:40:32

文章被收录于专栏：搜狗测试搜狗测试

前言

基于Jenkins的服务端持续集成已在搜狗商业产品系统实现，实施流程如下图，今天介绍如何在服务端实施持续集成。

Jenkins工程配置

1.新建Jenkins Pipline工程

New Item -> Pipline

2.增加以下Params

ID	Tpye	Name	Description
1	UnitTest	Boolean Parameter	是否执行单元测试
2	Branch	Git Parameter	Git分支
3	RunSonar	Boolean Parameter	是否静态代码扫描
4	ApiTest	Boolean Parameter	是否需要接口测试
5	ServerIps	String Parameter	部署环境IP
6	RemoteU	String Parameter	部署用户名

3.填入Git地址和JenkinsFile名称

配置文件创建或修改

1.build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:jacoco="antlib:org.jacoco.ant" default="jacoco" name="Jacoco">
  <!--Jacoco 的安装路径,需要放在ant程序所在的目录，否则会报错-->
  <property name="jacocoantPath"
    value=""/>
  <!--最终生成 .exec 文件的路径，Jacoco 就是根据这个文件生成最终的报告的-->
  <property name="integrationJacocoexecPath" value="./jacoco.exec"/>
  <!--合并后exec文件-->
  <property name="allJacocoexecPath" value=""/>
  <!--待合并的exec文件-->
  <property name="baseDir" value=""/>
  <!--生成覆盖率报告的路径-->
  <property name="reportfolderPath" value=""/>
  <!--源代码路径-->
  <property name="checkOrderSrcPath" value="src/main"/>
  <!--.class 文件路径-->
  <property name="checkOrderClasspath" value="build/classes"/>

  <!--让 ant 知道去哪儿找 Jacoco-->
  <target name="dump">
    <jacoco:dump address="" append="false"
      destfile="${integrationJacocoexecPath}"
      port="8044" reset="true"/>
  </target>

  <!--dump 任务:
             根据配置的 Ip 地址，和端口号，
      访问目标 Tomcat 服务，并生成 .exec 文件。-->
  <taskdef resource="org/jacoco/ant/antlib.xml" uri="antlib:org.jacoco.ant">
    <classpath path="${jacocoantPath}"/>
  </taskdef>
</project>

2.sonar-project.properties

sonar.projectKey=rtbmanager:1.0
# this is the name and version displayed in the SonarQube UI. Was mandatory prior to SonarQube 6.1.
sonar.projectName=rtbmanager-1.0
sonar.projectVersion=1.0

# Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows.
# This property is optional if sonar.modules is set.
sonar.sources=src
sonar.exclusions=**/test/**,**/target/**
sonar.java.binaries=build/classes/
sonar.java.source=1.8
sonar.java.target=1.8
# Encoding of the source code. Default is default system encoding
sonar.sourceEncoding=UTF-8
# Set jacoco Configuration
# coverage
sonar.core.codeCoveragePlugin=jacoco
# exec binary files
sonar.jacoco.reportPaths=build/jacoco-ut.exec
# Path to the JaCoCo report file containing coverage data by integration tests（集成测试）. 
# The path may be absolute or relative to the project base directory
sonar.jacoco.itReportPath=cifiles/jacoco.exec
# additional
sonar.dynamicAnalysis=reuseReports
#sonar.jacoco.reportMissing.force.zero=false
sonar.coverage.exclusions=**/rtb/manager/config/**

3.Java配置文件dev.conf

JAVA_OPTS="-javaagent:/search/odin/jacocoagent.jar=includes=com.sogou.*,output=tcpserver,port=8044,address=127.0.0.1,append=true -Xverify:none"

4.build.gradle/pom.xml

build.gradle

//Jacoco
apply plugin: "jacoco"
 
jacoco {
    toolVersion = "0.8.3"
    reportsDir = file("$buildDir/customJacocoReportDir")
}
jacocoTestReport {
    reports {
        xml.enabled false
        csv.enabled false
        html.destination file("${buildDir}/jacocoHtml")
    }
}
test {
    jacoco {
        destinationFile = file("$buildDir/jacoco-ut.exec")
        classDumpDir = file("$buildDir/classpathdumps")
    }
}

Jenkins Pipline文件修改

1.Build Stage修改

对于gradle工程来说，单元测试的执行在编译过程就会执行。

./gradlew build -Pprofile=${profile}

该命令即可在编译过程执行单元测试，单元测试通过编译成功，反之失败。

2.UnitTest Stage修改

该stage用于单元测试代码覆盖率统计。修改classPattern参数，改为对应工程需要统计覆盖率类的目录。

3.SonarQube Scan Stage

该stage将编译后的程序提交至SonarQube，并根据SonarQube返回的结果判定该本次pipline的执行是否成功 SonarQube Scanner的使用方式有两种，

Jenkins插件模式

已安装SonarQube Scanner插件

def scannerHome = tool 'sonarqube_scanner';       sh "${scannerHome}/bin/sonar-scanner -D project.settings=cifiles/sonar-project.properties"

Jenkins机器手动安装SonarQube Scanner程序（需要在Jenkins机器安装SonarQube Scanner）

sh '/search/odin/sonar/sonar-scanner-cli/bin/sonar-scanner  -D project.settings=cifiles/sonar-project.properties'

4.ApiTest Stage

该stage 用于执行接口自动化用例，同时统计其覆盖率，并与单元测试覆盖率合并，最终的覆盖率结果在SonarQube上展现

1.修改build job: '{project}-apitest'

2.修改Ant执行方式

Ant的使用方式有两种，插件模式和手动安装模式，更推荐插件模式，以下是两种模式的代码信息

插件模式

withAnt(installation: 'ant'){
         sh 'ant dump -buildfile cifiles/build.xml'
     }

直接安装

sh '/usr/local/software/apache-ant-1.9.14/bin/ant dump -buildfile cifiles/build-dispatcher.xml'

Jenkins Pipline样例

#!groovy
node {
    def tomcat_health_suf = ':8080/actuator/health'
    def nginx_health_suf = ':80/actuator/health'
    def max_secs = 30
    stage('git clone') {
        checkout scm
    }
 
    stage('gradle build') {
        sh './gradlew build -Pprofile=${profile}'
    }

    stage('UnitTest'){
        if (unittest == 'true'){
            echo "deploy jacoco......"
            for (ip in serverIps.split(',')) {
                if (deployJacoco(ip, remote_user) == false) {
                    echo '[FAILURE] Failed to deploy jacoco'
                    currentBuild.result = 'FAILURE'
                    return
                }
            }
 
            echo "starting unitTest jacocoCoverage......"
            /*jacoco changeBuildStatus: true, maximumLineCoverage:"80"*/
            jacoco(
                    buildOverBuild: false,
                    changeBuildStatus: false,
                    execPattern: 'build/jacoco-ut.exec',
                    classPattern: 'build/classese',
                    sourcePattern: 'src/main/java',
                    exclusionPattern: 'src/test*',
                    minimumMethodCoverage: '10',
                    maximumMethodCoverage: '90',
                    minimumClassCoverage: '40',
                    maximumClassCoverage: '90',
                    minimumLineCoverage: '5',
                    maximumLineCoverage: '90'
            )
        }
    }
 
    stage('SonarQube Scan') {
        if (runSonar == 'true') {
            withSonarQubeEnv('sonarqube') {
                //注意这里withSonarQubeEnv()中的参数要与之前SonarQube servers中Name的配置相同
                echo "starting codeAnalyze with SonarQube......"
                echo "WORKSPACE: $WORKSPACE"
                //执行Sonar Scanner
                def scannerHome = tool 'sonarqube_scanner';
                sh "${scannerHome}/bin/sonar-scanner -D project.settings=cifiles/sonar-project.properties"
                // sh '/search/odin/sonar/sonar-scanner-cli/bin/sonar-scanner  -D project.settings=cifiles/sonar-project.properties'
            }
            script {
                timeout(1) {
                    //这里设置超时时间1分钟，如果Sonar Webhook失败，不会出现一直卡在检查状态
                    //利用Sonar webhook功能通知pipeline代码检测结果，未通过质量阈，pipeline将会fail
                    def qg = waitForQualityGate('sonarqube')
                    //注意：这里waitForQualityGate()中的参数也要与之前SonarQube servers中Name的配置相同
                    if (qg.status != 'OK') {
                        error "未通过Sonarqube的代码质量阈检查，请及时修改！failure: ${qg.status}"
                    }
                }
            }
        }
    }
 
    if (confirming == 'true') {
        stage('staging') {
            if (deploy(stagingIp, remote_user, "staging", tomcat_health_suf, nginx_health_suf, max_secs, nginx) == false) {
                echo '[FAILURE] Failed to build'
                currentBuild.result = 'FAILURE'
                return
            }
        }
    }
 
    stage('deploy') {
        if (confirming == 'true') {
            input 'make sure to publish?'
        }
        for (ip in serverIps.split(',')) {
            if (deploy(ip, remote_user, profile, tomcat_health_suf, nginx_health_suf, max_secs, nginx) == false) {
                echo '[FAILURE] Failed to build'
                currentBuild.result = 'FAILURE'
                return
            }
        }
    }
 
    stage('ApiTest'){
        if (apiTest == 'true') {
            build job: 'rtbmanager-apitest'
            echo 'starting ant dump......'
            withAnt(installation: 'ant'){
              sh 'ant dump -buildfile cifiles/build.xml'
            }
            // sh '/usr/local/software/apache-ant-1.9.14/bin/ant dump -buildfile /usr/local/software/apache-ant-1.9.14/build-rtbmanager.xml'
 
            echo 'starting apiTest jacocoCoverage......'
            /*jacoco changeBuildStatus: true, maximumLineCoverage:"80"*/
            jacoco(
                    buildOverBuild: false,
                    changeBuildStatus: false,
                    execPattern: 'cifiles/jacoco.exec',
                    classPattern: 'build/classes',
                    sourcePattern: 'src/main/java',
                    exclusionPattern: 'src/test*',
                    minimumMethodCoverage: '70',
                    maximumMethodCoverage: '90',
                    minimumClassCoverage: '80',
                    maximumClassCoverage: '90',
                    minimumLineCoverage: '60',
                    maximumLineCoverage: '90'
            )
 
            withSonarQubeEnv('sonarqube') {
                //注意这里withSonarQubeEnv()中的参数要与之前SonarQube servers中Name的配置相同
                echo "starting codeAnalyze with SonarQube......"
                //统计接口测试覆盖率，并同步至SonarQube
                def scannerHome = tool 'sonarqube_scanner';
                sh "${scannerHome}/bin/sonar-scanner -D project.settings=cifiles/sonar-project.properties"
            }
        }
    }
}
 
def deploy(ip, remote_user, profile, tomcat_health_suf, nginx_health_suf, max_secs, nginx) {
    if (nginx == 'true') {
        sh "ssh ${remote_user}@${ip} \"sudo systemctl stop openresty.service\""
    }
    sh "scp build/libs/app.jar ${remote_user}@${ip}:/search/odin/app/app.jar"
    sh "scp envfiles/${profile}.conf ${remote_user}@${ip}:/search/odin/app/app.conf"
    sh "ssh ${remote_user}@${ip} \"sudo systemctl restart myapp.service\""
    if (check(ip + tomcat_health_suf, max_secs, 8) == false) {
        return false
    }
    if (nginx == 'true') {
        sh "ssh ${remote_user}@${ip} \"sudo systemctl start openresty.service\""
        return check(ip + nginx_health_suf, max_secs, 1)
    }
    return true
}
 
 
def deployJacoco(ip, remote_user) {
    sh "scp -r cifiles/jacocoagent.jar ${remote_user}@${ip}:/search/odin/"
    sh "ssh ${remote_user}@${ip} \"sudo systemctl restart myapp.service\""
    return true
}
 
def check(url, max, initial_sleep_secs) {
    if (!url.startsWith("http")) {
        url = 'http://' + url
    }
    def cmd = $/eval "curl -s ${url} | sed 's/ //g' | grep '\"status\":\"UP\"' | wc -l"/$
    echo "${cmd}"
    def rc = "0";
    sleep(initial_sleep_secs); // seconds
    try {
        rc = sh(
                script: "${cmd}",
                returnStdout: true
        ).trim();
        echo rc
    } catch (Exception e) {
    }
    def i = 0;
    while (rc.equals("0") && i < max) {
        sleep(1); // seconds
        try {
            rc = sh(
                    script: "${cmd}",
                    returnStdout: true
            ).trim();
            echo rc
        } catch (Exception e) {
        }
        i++;
    }
    echo rc
    return rc.equals("0") ? false : true;
}

服务端持续集成效果展示

Jenkins持续集成构建结果：