前言
什么是误报? 这里定义为不是用例本身错误导致的错误(网络、环境等原因造成的失败)。
对于自动化接口测试之前也没有太多的研究,经历了一些实践。先通过Jenkins完成自动化测试框架的搭建和集成(ant+jmeter),然后不断优化将jmeter 纳入到了容器中。其大概过程是通过ant批量的执行测试脚本,最后在jenkins中生成测试报告
关于测试报告中的内容包含以下信息,成功和失败用例的信息。 我们先根据接口的返回过滤出一些误报的失败用例(类似于网络错误、环境问题等)。然后通过Jenkins的交互的展示给用户,供用户再次选择其他接口是否也属于误报。确认后将误报的用例计入成功用例数,重新计算成功率和失败率以及误报率。(这肯定不是最佳的做法,最佳的做法应该是有一个测试的平台或系统,当前做法也算是可以完成误报率的统计)
准备实践
需要准备内容
生成测试报告
打开jmeter然后创建一个测试用例(这里故意将地址写错)
资源地址:https://github.com/zeyangli/jmetertest.git
运行测试
ZeyangdeMacBook-Pro:jmetertest zeyang$ ls
README.md jmeter.results.shanhe.me.xsl
build.xml result
jmeter-results-detail-report_21.xsl scripts
jmeter.log
ZeyangdeMacBook-Pro:jmetertest zeyang$ ant -buildfile ./build.xml
Buildfile: /Users/zeyang/Desktop/jmetertest/build.xml
run:
test:
[jmeter] Executing test plan: /Users/zeyang/Desktop/jmetertest/scripts/blog.jmx ==> /Users/zeyang/Desktop/jmetertest/result/jtlfile/TestReport_201911241112.jtl
[jmeter] 2019-11-24 11:12:31,449 main ERROR FileManager (jmeter.log) java.io.FileNotFoundException: jmeter.log (Permission denied) java.io.FileNotFoundException: jmeter.log (Permission denied)
[jmeter] Creating summariser <summary>
[jmeter] Created the tree successfully using /Users/zeyang/Desktop/jmetertest/scripts/blog.jmx
[jmeter] Starting the test @ Sun Nov 24 11:12:31 CST 2019 (1574565151991)
[jmeter] Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
[jmeter] summary + 1 in 00:00:00 = 4.3/s Avg: 34 Min: 34 Max: 34 Err: 1 (100.00%) Active: 3 Started: 3 Finished: 0
[jmeter] summary + 3 in 00:00:01 = 3.9/s Avg: 338 Min: 0 Max: 807 Err: 1 (33.33%) Active: 0 Started: 3 Finished: 3
[jmeter] summary = 4 in 00:00:01 = 4.0/s Avg: 262 Min: 0 Max: 807 Err: 2 (50.00%)
[jmeter] Tidying up ... @ Sun Nov 24 11:12:33 CST 2019 (1574565153497)
[jmeter] ... end of run
report:
[xslt] Processing /Users/zeyang/Desktop/jmetertest/result/jtlfile/TestReport_201911241112.jtl to /Users/zeyang/Desktop/jmetertest/result/htmlfile/201911241112_DetailReport.html
[xslt] Loading stylesheet /usr/local/apache-jmeter-5.1.1/extras/jmeter.results.shanhe.me.xsl
[xslt] Processing /Users/zeyang/Desktop/jmetertest/result/jtlfile/TestReport_201911241112.jtl to /Users/zeyang/Desktop/jmetertest/result/htmlfile/201911241112_SummaryReport.html
[xslt] Loading stylesheet /usr/local/apache-jmeter-5.1.1/extras/jmeter-results-detail-report_21.xsl
BUILD SUCCESSFUL
Total time: 4 seconds
查看测试报告效果
找到生成的测试报告,上传到github。(非必须上传github)
后面就需要Jenkins去下载测试报告并解析了
解析测试报告
准备通过groovy 正则表达式完成HTML报告的分析,发现共同特征。
编写Jenkinsfile
#!groovy
def falsePositive = []
node {
stage("GetCode"){
//checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '24982560-17fc-4589-819b-bc5bea89da77', url: 'https://github.com/zeyangli/jmetertest.git']]])
sh " curl https://raw.githubusercontent.com/zeyangli/jmetertest/master/SummaryReport.html -o SummaryReport.html"
sh "cat SummaryReport.html"
publishHTML([allowMissing: false, alwaysLinkToLastBuild: false, keepAll: false, reportDir: '', reportFiles: 'SummaryReport.html', reportName: '接口测试报告', reportTitles: ''])
}
stage("fenxi"){
def fileContext = readFile "./SummaryReport.html"
println(fileContext)
def pattern = "<h3>.*?<a"
def matcher = fileContext =~ pattern
if (matcher.find()){
println(matcher)
matcher.each {
println(it - "<h3>" - "<a")
falseCase = it - "<h3>" - "<a"
falsePositive.add(falseCase)
}
}
println(falsePositive)
}
stage("UserAccept"){
def result = input message: '请选择属于误报的用例',
ok: '提交',
parameters: [extendedChoice( description: '误报的失败用例',
descriptionPropertyValue: '',
multiSelectDelimiter: ',',
name: 'failePositiveCases',
quoteValue: false,
saveJSONParameterToFile: false,
type: 'PT_CHECKBOX',
value: "${falsePositive.toString() - "[" - ']'}",
visibleItemCount: 99)]
println(result)
}
}
运行效果
FAQ: 关于 extendedChoice 参数的使用
完成上述操作后,就可以计算本次测试中的误报用例数量。并将误报数据收集到度量系统中。