Jmeter Redis插件开发-读写数据

背景

最近一段时间在接触性能压测,遇到一个棘手的问题。性能需求在30KQPS,要求进行单接口压测,接口之间依赖不可避免(下一个接口发压数据需要使用上一接口的返回),还不能通过做数据的方式准备。只能将上一接口返回的数据,保存起来,用于下一接口的参数。

在一开始的时候,犯了一个很二的错误,将数据写入到Jmeter的日志中,再进行提取(发压端文件IO影响性能不是一点点),然后将受影响的性能指标作为测试结果(可进行两次测试,第一次,不写日志,性能指标作为测试结果,第二次,写日志,采集的数据作为下一接口的配置元)。

当然了,这样的工作,为什么要每次都做两遍呢,当然可以考虑开发一个插件来做了。首先考虑的就是写内存记录下来。

主要考虑两个方向:

① Jmeter内存

② Redis

其中,Jmeter内存保存,仅在本次测试有效,还需要在测试结束时,将内存中数据,保存到文件。考虑用redis,网上搜索了一把,jmeter-plugins-redis能满足读取redis数据的需求(不重复造轮子,就用它了),只要自己完成一个写入数据到redis的插件就可以了。

准备工作

下载Redis插件

官方提供的网址:https://jmeter-plugins.org/wiki/RedisDataSet/

下载下来之后,继续下载依赖的jedis版本,官方依赖的是jedis 2.2.1

然鹅,在jmeter 3.0版本下,竟然用不起来,抛出了一票的异常。度娘了一把(请原谅,公司翻墙要自备梯子),是jedis的版本过低,jedis 2.4.1 以上解决了。正好,项目组其他插件依赖2.8.1,索性就用这个版本。

源码修改

用Jedis 2.8.1后,又抛异常,真是沮丧啊。怎么办?思来想去,改源码,就不信这个邪!github上一搜索,还真就有。二话不说,下载源码:https://github.com/undera/jmeter-plugins,这位老哥undera,还是很厚道的,插件源码都来了。

打开源码,进行编译,提示下面3个函数已经没有了:

那就看下怎么修改吧,setMaxActive -> setMaxTotal,setMaxWait -> setMaxWaitMillis,最后一个实在没有找到,先注释掉。

编译,生成jar包,放进去跑一把,界面出来了。读取数据,就算搞定了,下面去开发写数据。

二次开发

由于前面已经下载了源码,而且jmeter插件网站上,提供了源码,怎么配合Redis Data Set的get mode,插入数据。

那就先做简单的,证明这种方式可以用。考虑用后置处理器来解决。

简单应用

1、添加数据操作类:RedisDataWriter

2、添加界面显示:RedisDataWriterGUI

想法很简单,先用起来,能在Jmeter上,找到这个后置处理器,能将数据插入到redis里面。连接信息之类的,都先给写死。

跑完第一把,发现数据竟然能写到redis,很满足了。

添加界面

有了上面的经验,那就可以慢慢的往上添加界面,优化连接。

RedisDataWriterGUI.java

4.

5.importjava.awt.*;

6.

7.importjava.util.List;

8.

9.importjavax.swing.BorderFactory;

10.importjavax.swing.Box;

11.importjavax.swing.JComponent;

12.importjavax.swing.JPanel;

13.

18.

19.publicclassRedisDataWriterGUIextendsAbstractPostProcessorGui{

20.

21.privateJLabeledTextField redisKeyField;

22.privateJLabeledTextField variableNamesField;

23.privateJLabeledChoice addModeField;

24.

25.privateJLabeledTextField redisServerHostField;

26.privateJLabeledTextField redisServerPortField;

27.privateJLabeledTextField redisServerTimeoutField;

28.privateJLabeledTextField redisServerPasswdField;

29.privateJLabeledTextField redisServerDbField;

30.

31.privateJLabeledTextField redisPoolMinIdleField;

32.privateJLabeledTextField redisPoolMaxIdleField;

33.privateJLabeledTextField redisPoolMaxActiveField;

34.privateJLabeledTextField redisPoolMaxWaitField;

35.

36.publicRedisDataWriterGUI() {

37.super();

38.init();

39.}

40.

41.@Override

42.publicvoidconfigure(TestElement el) {

43.super.configure(el);

44.if(elinstanceofRedisDataWriter){

45.RedisDataWriter re = (RedisDataWriter) el;

46.redisKeyField.setText(re.getRedisKey());

47.variableNamesField.setText(re.getVariableNames());

48.addModeField.setText(re.getRedisAddMode());

49.redisServerHostField.setText(re.getRedisHost());

50.redisServerPortField.setText(re.getRedisPort());

51.redisServerTimeoutField.setText(re.getRedisTimeout());

52.redisServerPasswdField.setText(re.getRedisPasswd());

53.redisServerDbField.setText(re.getRedisDatabase());

54.redisPoolMinIdleField.setText(Integer.toString(re.getRedisMinIdle()));

55.redisPoolMaxIdleField.setText(Integer.toString(re.getRedisMaxIdle()));

56.redisPoolMaxActiveField.setText(Integer.toString(re.getRedisMaxActive()));

57.redisPoolMaxWaitField.setText(Integer.toString(re.getRedisMaxWait()));

58.

59.redisKeyField.setLabel("Redis key: ");

60.variableNamesField.setLabel("Redis value (可以是变量列表,逗号分隔,例如:$,$): ");

61.addModeField.setLabel("Redis写入模式: ");

62.redisServerHostField.setLabel("Redis host: ");

63.redisServerPortField.setLabel("Redis port: ");

64.redisServerTimeoutField.setLabel("连接超时设置(ms): ");

65.redisServerPasswdField.setLabel("连接密码: ");

66.redisServerDbField.setLabel("数据库: ");

67.redisPoolMinIdleField.setLabel("minIdle: ");

68.redisPoolMaxIdleField.setLabel("maxIdle: ");

69.redisPoolMaxActiveField.setLabel("maxActive: ");

70.redisPoolMaxWaitField.setLabel("maxWait(s):");

71.}

72.}

73.

74./**

75.* Implements JMeterGUIComponent.clearGui

76.*/

77.@Override

78.publicvoidclearGui() {

79.super.clearGui();

80.

81.redisKeyField.setText("");//$NON-NLS-1$

82.variableNamesField.setText("");//$NON-NLS-1$

83.addModeField.setText("");

84.redisServerHostField.setText("");//$NON-NLS-1$

85.redisServerPortField.setText("");//$NON-NLS-1$

86.redisServerTimeoutField.setText("");

87.redisServerPasswdField.setText("");

88.redisServerDbField.setText("");

89.redisPoolMinIdleField.setText("");

90.redisPoolMaxIdleField.setText("");

91.redisPoolMaxActiveField.setText("");

92.redisPoolMaxWaitField.setText("");

93.}

94.

95.@Override

96.publicTestElement createTestElement() {

97.// TODO Auto-generated method stub

98.RedisDataWriter extractor =newRedisDataWriter();

99.modifyTestElement(extractor);

100.returnextractor;

101.}

102.

103.@Override

104.publicString getLabelResource() {

105.// TODO Auto-generated method stub

106.returnthis.getClass().getName();

107.}

108.

109.@Override

110.publicString getStaticLabel() {//设置显示名称

111.// TODO Auto-generated method stub

112.return"Redis数据录入器";

113.}

114.

115.@Override

116.publicvoidmodifyTestElement(TestElement extractor) {

117.// TODO Auto-generated method stub

118.super.configureTestElement(extractor);

119.if(extractorinstanceofRedisDataWriter) {

120.RedisDataWriter r = (RedisDataWriter) extractor;

121.r.setRediskey(redisKeyField.getText());

122.r.setVariableNames(variableNamesField.getText());

123.r.setRedisAddMode(addModeField.getText());

124.r.setRedisHost(redisServerHostField.getText());

125.r.setRedisPort(redisServerPortField.getText());

126.r.setRedisTimeout(redisServerTimeoutField.getText());

127.r.setRedisPasswd(redisServerPasswdField.getText());

128.r.setRedisDatabase(redisServerDbField.getText());

129.r.setRedisMinIdle(Integer.parseInt(redisPoolMinIdleField.getText()));

130.r.setRedisMaxIdle(Integer.parseInt(redisPoolMaxIdleField.getText()));

131.r.setRedisMaxActive(Integer.parseInt(redisPoolMaxActiveField.getText()));

132.r.setRedisMaxWait(Integer.parseInt(redisPoolMaxWaitField.getText()));

133.}

134.

135.}

136.

137.privatevoidinit() {

138.setLayout(newBorderLayout());

139.setBorder(makeBorder());

140.Box box = Box.createVerticalBox();

141.box.add(makeTitlePanel());

142.box.add(makeRedisDataPanel());

143.add(box, BorderLayout.NORTH);

144.box.add(makeRedisConnectionPanel());

145.box.add(makeRedisPoolPanel());

146.}

147.

148.privateJPanel makeRedisDataPanel() {

149.redisKeyField =newJLabeledTextField(JMeterUtils.getResString("rediskey_field"));//$NON-NLS-1$

150.variableNamesField =newJLabeledTextField(JMeterUtils.getResString("variable_names_field"));//$NON-NLS-1$

151.addModeField =newJLabeledChoice(JMeterUtils.getResString("add_mode_field"),false);

152.addModeField.addValue("LST_RPUSH");

153.addModeField.addValue("SET_ADD");

154.

155.JPanel panel =newJPanel(newGridBagLayout());

156.panel.setBorder(BorderFactory.createTitledBorder("Redis数据配置"));//$NON-NLS-1$

157.GridBagConstraints gbc =newGridBagConstraints();

158.initConstraints(gbc);

159.addField(panel, redisKeyField, gbc);

160.resetContraints(gbc);

161.addField(panel, variableNamesField, gbc);

162.resetContraints(gbc);

163.addField(panel, addModeField, gbc);

164.

165.returnpanel;

166.}

167.

168.privateJPanel makeRedisConnectionPanel() {

169.redisServerHostField =newJLabeledTextField(JMeterUtils.getResString("redis_server_host_field"));//$NON-NLS-1$

170.redisServerPortField =newJLabeledTextField(JMeterUtils.getResString("redis_server_port_field"));//$NON-NLS-1$

171.redisServerTimeoutField =newJLabeledTextField(JMeterUtils.getResString("redis_server_timeout_field"));

172.redisServerPasswdField =newJLabeledTextField(JMeterUtils.getResString("redis_server_passwd_field"));

173.redisServerDbField =newJLabeledTextField(JMeterUtils.getResString("redis_server_database_field"));

174.

175.JPanel panel =newJPanel(newGridBagLayout());

176.panel.setBorder(BorderFactory.createTitledBorder("Redis连接配置"));//$NON-NLS-1$

177.GridBagConstraints gbc =newGridBagConstraints();

178.initConstraints(gbc);

179.addField(panel, redisServerHostField, gbc);

180.resetContraints(gbc);

181.addField(panel, redisServerPortField, gbc);

182.resetContraints(gbc);

183.addField(panel, redisServerTimeoutField, gbc);

184.resetContraints(gbc);

185.addField(panel, redisServerPasswdField, gbc);

186.resetContraints(gbc);

187.addField(panel, redisServerDbField, gbc);

188.

189.returnpanel;

190.}

191.

192.privateJPanel makeRedisPoolPanel() {

193.redisPoolMinIdleField =newJLabeledTextField(JMeterUtils.getResString("redis_pool_minidle"));

194.redisPoolMaxIdleField =newJLabeledTextField(JMeterUtils.getResString("redis_pool_maxidle"));

195.redisPoolMaxActiveField =newJLabeledTextField(JMeterUtils.getResString("redis_pool_maxactive"));

196.redisPoolMaxWaitField =newJLabeledTextField(JMeterUtils.getResString("redis_pool_maxwait"));

197.

198.JPanel panel =newJPanel(newGridBagLayout());

199.panel.setBorder(BorderFactory.createTitledBorder("Redis连接池配置"));//$NON-NLS-1$

200.GridBagConstraints gbc =newGridBagConstraints();

201.initConstraints(gbc);

202.addField(panel, redisPoolMinIdleField, gbc);

203.resetContraints(gbc);

204.addField(panel, redisPoolMaxIdleField, gbc);

205.resetContraints(gbc);

206.addField(panel, redisPoolMaxActiveField, gbc);

207.resetContraints(gbc);

208.addField(panel, redisPoolMaxWaitField, gbc);

209.

210.returnpanel;

211.}

212.

213.privatevoidaddField(JPanel panel, JLabeledField field, GridBagConstraints gbc) {

214.List item = field.getComponentList();

215.panel.add(item.get(), gbc.clone());

216.gbc.gridx++;

217.gbc.weightx =1;

218.gbc.fill=GridBagConstraints.HORIZONTAL;

219.panel.add(item.get(1), gbc.clone());

220.}

221.

222.// Next line

223.privatevoidresetContraints(GridBagConstraints gbc) {

224.gbc.gridx =;

225.gbc.gridy++;

226.gbc.weightx =;

227.gbc.fill=GridBagConstraints.NONE;

228.}

229.

230.privatevoidinitConstraints(GridBagConstraints gbc) {

231.gbc.anchor = GridBagConstraints.NORTHWEST;

232.gbc.fill = GridBagConstraints.NONE;

233.gbc.gridheight =1;

234.gbc.gridwidth =1;

235.gbc.gridx =;

236.gbc.gridy =;

237.gbc.weightx =;

238.gbc.weighty =;

239.}

240.}

RedisDataWriter.java

1./*

2.* Created on 2018/01/26

3.* @author: ben大神点C

4.*/

5.

7.

8.importjava.io.Serializable;

9.

18.

23.

24.

25.publicclassRedisDataWriterextendsAbstractScopedTestElementimplementsPostProcessor, Serializable, TestStateListener {

26.privatestaticfinalLogger log = LoggingManager.getLoggerForClass();

27.

28.privatestaticfinalString REDISKEY ="RedisData.redisKey";// $NON-NLS-1$

29.privatestaticfinalString VARIABLE_NAMES ="RedisData.variableNames";

30.privatestaticfinalString REDISADDMODE ="RedisData.addMode";

31.privatestaticfinalString REDISHOST ="RedisServer.host";

32.privatestaticfinalString REDISPORT ="RedisServer.port";

33.privatestaticfinalString REDISTIMEOUT ="RedisServer.timeout";

34.privatestaticfinalString REDISPASSWD ="RedisServer.passwd";

35.privatestaticfinalString REDISDATABSE ="RedisServer.database";

36.privatestaticfinalString REDISMINIDLE ="RedisPool.minIdle";

37.privatestaticfinalString REDISMAXIDLE ="RedisPool.maxIdle";

38.privatestaticfinalString REDISMAXACTIVE ="RedisPool.maxActive";

39.privatestaticfinalString REDISMAXWAIT ="RedisPool.maxWait";

40.

41.privatetransientJedisPool pool;

42.

43.publicenumAddMode {

44.LST_RPUSH((byte)),

45.SET_ADD((byte)1);

46.privatebytevalue;

47.privateAddMode(bytevalue) {

48.this.value = value;

49.}

50.

51.publicbytegetValue() {

52.returnvalue;

53.}

54.}

55.

56.@Override

57.publicvoidtestStarted(String distributedHost) {

58.JedisPoolConfig config =newJedisPoolConfig();

59.config.setMaxTotal(getRedisMaxActive());

60.config.setMaxIdle(getRedisMaxIdle());

61.config.setMinIdle(getRedisMinIdle());

62.config.setMaxWaitMillis(getRedisMaxWait()*1000);

63.

64.String host = getRedisHost();

65.

66.intport = Protocol.DEFAULT_PORT;

67.if(!JOrphanUtils.isBlank(getRedisPort())) {

68.port = Integer.parseInt(getRedisPort());

69.}

70.inttimeout = Protocol.DEFAULT_TIMEOUT;

71.if(!JOrphanUtils.isBlank(getRedisTimeout())) {

72.timeout = Integer.parseInt(getRedisTimeout());

73.}

74.intdatabase = Protocol.DEFAULT_DATABASE;

75.if(!JOrphanUtils.isBlank(getRedisDatabase())) {

76.database = Integer.parseInt(getRedisDatabase());

77.}

78.String password =null;

79.if(!JOrphanUtils.isBlank(getRedisPasswd())) {

80.password = getRedisPasswd();

81.}

82.this.pool =newJedisPool(config, host, port, timeout, password, database);

83.//System.out.println("testStarted:" + this.pool + "this:" + this);

84.}

85.

86.@Override

87.publicvoidtestEnded() {

88.testEnded("");

89.}

90.

91.@Override

92.publicvoidtestEnded(String host) {

93.pool.destroy();

94.}

95.

96.@Override

97.publicvoidtestStarted() {

98.testStarted("");

99.}

100.

101.@Override

102.publicObject clone() {

103.RedisDataWriter clonedElement = (RedisDataWriter)super.clone();

104.clonedElement.pool =this.pool;

105.returnclonedElement;

106.}

107.

108.

109.@Override

110.publicvoidprocess() {

111.// TODO Auto-generated method stub

112.JMeterContext context = getThreadContext();

113.SampleResult previousResult = context.getPreviousResult();

114.if(previousResult ==null) {

115.return;

116.}

117.log.debug("RedisDataWriter processing result");

118.

119.try{

120.String redisKey = getRedisKey();

121.String redisValue = getVariableNames();

122.

124.//System.out.println("testStarted:" + this.pool + "this:" + this);

125.

126.try{

127.Enum mode = getAddMode();

128.if(mode.equals(AddMode.LST_RPUSH)) {

129.connection.rpush(redisKey, redisValue);

130.}else{

131.connection.sadd(redisKey, redisValue);

132.}

133.}finally{

134.if(connection !=null) {

136.}

137.}

138.

139.}catch(Exception e) {

140.e.printStackTrace();

141.}

142.

143.}

144.

145.publicvoidsetRediskey(String redisKey) {

146.setProperty(REDISKEY, redisKey);

147.}

148.

149.publicString getRedisKey() {

150.returngetPropertyAsString(REDISKEY);

151.}

152.

153.publicvoidsetVariableNames(String variableNames) {

154.setProperty(VARIABLE_NAMES, variableNames);

155.}

156.

157.publicString getVariableNames() {

158.returngetPropertyAsString(VARIABLE_NAMES);

159.}

160.

161.publicvoidsetRedisHost(String host) {

162.setProperty(REDISHOST, host);

163.}

164.

165.publicString getRedisHost() {

166.returngetPropertyAsString(REDISHOST);

167.}

168.

169.publicvoidsetRedisPort(String port) {

170.setProperty(REDISPORT, port);

171.}

172.

173.publicString getRedisPort() {

174.String port = getPropertyAsString(REDISPORT);

175.if(JOrphanUtils.isBlank(port)) {

176.port = Integer.toString(Protocol.DEFAULT_PORT);

177.}

178.returnport;

179.}

180.

181.publicvoidsetRedisTimeout(String timeout) {

182.setProperty(REDISTIMEOUT, timeout);

183.}

184.

185.publicString getRedisTimeout() {

186.String timeout = getPropertyAsString(REDISTIMEOUT);

187.if(JOrphanUtils.isBlank(timeout)) {

188.timeout = Integer.toString(Protocol.DEFAULT_TIMEOUT);

189.}

190.returntimeout;

191.}

192.

193.publicvoidsetRedisPasswd(String password) {

194.setProperty(REDISPASSWD, password);

195.}

196.

197.publicString getRedisPasswd() {

198.returngetPropertyAsString(REDISPASSWD,null);

199.}

200.

201.publicvoidsetRedisDatabase(String db) {

202.setProperty(REDISDATABSE, db);

203.}

204.

205.publicString getRedisDatabase() {

206.String database = getPropertyAsString(REDISDATABSE);

207.if(JOrphanUtils.isBlank(database)) {

208.database = Integer.toString(Protocol.DEFAULT_DATABASE);

209.}

210.returndatabase;

211.}

212.

213.publicvoidsetRedisAddMode(String mode) {

214.for(Enum e : AddMode.values()) {

215.finalString propName = e.toString();

216.//System.out.println("propName:" + propName + ", mode:" + mode + ", name:" + e.name());

217.if(mode.equals(propName)) {

218.finalinttmpMode = e.ordinal();

219.if(log.isDebugEnabled()) {

220.log.debug("Converted "+"addMode="+ mode +" to mode="+ tmpMode);

221.}

222.super.setProperty(REDISADDMODE, e.toString());

223.return;

224.}

225.}

226.

227.super.setProperty(REDISADDMODE, AddMode.LST_RPUSH.ordinal());

228.}

229.

230.publicString getRedisAddMode() {

231.returngetPropertyAsString(REDISADDMODE, AddMode.LST_RPUSH.toString());

232.}

233.

234.publicEnum getAddMode() {

235.String mode = getRedisAddMode();

236.for(Enum e : AddMode.values()) {

237.finalString propName = e.toString();

238.if(mode.equals(propName)) {

239.returne;

240.}

241.}

242.returnAddMode.LST_RPUSH;

243.}

244.

245.publicvoidsetRedisMinIdle(intminIdle) {

246.setProperty(REDISMINIDLE, minIdle);

247.}

248.

249.publicintgetRedisMinIdle() {

250.returngetPropertyAsInt(REDISMINIDLE,);

251.}

252.

253.publicvoidsetRedisMaxIdle(intmaxIdle) {

254.setProperty(REDISMAXIDLE, maxIdle);

255.}

256.

257.publicintgetRedisMaxIdle() {

258.returngetPropertyAsInt(REDISMAXIDLE,10);

259.}

260.

261.publicvoidsetRedisMaxActive(intmaxActive) {

262.setProperty(REDISMAXACTIVE, maxActive);

263.}

264.

265.publicintgetRedisMaxActive() {

266.returngetPropertyAsInt(REDISMAXACTIVE,500);

267.}

268.

269.publicvoidsetRedisMaxWait(intmaxWait) {

270.setProperty(REDISMAXWAIT, maxWait);

271.}

272.

273.publicintgetRedisMaxWait() {

274.returngetPropertyAsInt(REDISMAXWAIT,30000);

275.}

276.

277.}

看下界面,基本上,需要设置的地方,都可以用了。

① 位置:Sampler -> 后置处理器:

② Rdis数据录入器:

优化界面

看了下代码,感觉还是有改进的可能性。起码界面这块,可以使用BenInfoSupport。

这里把代码连接贴一下:

https://github.com/eyotang/jmeter-plugins

写起来很简单,界面出起来也很快,还支撑多语言。最主要的是,和RedisDataSet放到同一个jar包里面。

测试性能

既然是性能测试使用,那插件本身也需要有一个性能指标吧。

这里使用本地的一个java程序产生的内容,使用Redis数据录入器写入单节点的redis中。

峰值:

QPS:62565.8 (6万),Redis的QPS已达到极限

CPU:33.2%, redis: 96%~97%, java:770%~780%

插件下载

为了让大家能直接下载到该版本的插件,这里将jedis-2.81和jmeter-plugins-redis都放进去了。

  • 发表于:
  • 原文链接:http://kuaibao.qq.com/s/20180201B0950J00?refer=cp_1026

扫码关注云+社区