Continuing with the series of blog posts regarding testing, automation and continuous integration, this time I will talk about how to integrate performance tests, in this case using Jmeter, with Maven and the Hudson continuous integration system. Jmeter is one of the main tools we use in our projects to create relevant performance tests, and automation and integration in our CI systems is essential. We also use SoapUI and Grinder depending on the platform, but we will cover those in future posts.
To integrate Jmeter and Maven, you must use the Maven Jmeter plugin, the first version of which is officially hosted here. There are are several posts discussing the use of this plugin that I have used as reference, particularly this one from James Lorenzen.
An updated version of this original plugin was released in google code by Ronald Alleva. It adds support for the latest version of Jmeter (2.3.2) and contains some additional enhacements such as parameterisation support. I decided to use this version to get this extra funcionality.
To make this post as simple and easy to follow as possible, my idea was to test Jmeter tests automation and integration against an application web running in an embeded Jetty container.
My objectives were:
- Be able to run my Jmeter tests in a specific maven phase.
- Create the relevant Jmeter test results html reports and publish them during the maven site phase.
- Integrate Jmeter execution and reporting with Hudson, via the Hudson Jmeter plugin available here.
This post is a combination of James and Ron’s posts, adding solutions to problems I discovered while creating and executing my tests and showing the integration with Hudson.
Ronald describes pretty well here the different steps required in order to make the plugin work. I will not include these steps but rather comment on them.
- Download the support jars and the maven Jmeter plugin and install them.
- Add the dependency to your maven project. In our case, we want to execute our Jmeter tests when we start the embedded jetty container. We usually start/stop Jetty as part of the pre-integration and post-integration maven phases. We just need to add the Jmeter plugin execution to the integration-test phase (I have also included the jetty part below):
<profiles> <profile> <id>env-dev-jetty</id> <activation> <activeByDefault>true</activeByDefault> </activation> <build> <plugins> <plugin> <groupId>org.apache.jmeter</groupId> <artifactId>maven-jmeter-plugin</artifactId> <version>1.0</version> <executions> <execution> <id>jmeter-tests</id> <phase>integration-test</phase> <goals> <goal>jmeter</goal> </goals> </execution> </executions> <configuration> <reportDir> ${project.build.directory}/jmeter-reports </reportDir> <jmeterUserProperties> <hostname>localhost</hostname> <port>9080</port> </jmeterUserProperties> </configuration> </plugin> <plugin> <groupId>org.mortbay.jetty</groupId> <artifactId>maven-jetty-plugin</artifactId> <version>6.1.12rc1</version> <configuration> <webApp> ${basedir}/target/${project.build.finalName}/${project.parent.name}-webapp-${project.version}.war </webApp> <contextPath> ${project.parent.name}-webapp </contextPath> <connectors> <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector"> <port>9080</port> <maxIdleTime>60000</maxIdleTime> </connector> </connectors> <systemProperties> <systemProperty> <name> pmtool.application.environment </name> <value>dev</value> </systemProperty> </systemProperties> <stopPort>9966</stopPort> <stopKey>foo</stopKey> <daemon>true</daemon> <scanIntervalSeconds>0</scanIntervalSeconds> </configuration> <executions> <execution> <id>start-jetty</id> <phase>pre-integration-test</phase> <goals> <goal>run-war</goal> </goals> </execution> <execution> <id>stop-jetty</id> <phase>post-integration-test</phase> <goals> <goal>stop</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </profile> </profiles>
In a real scenario where, as part of your project cycle, the application is deployed to a development or pre-production environment, we could use the verify phase to execute the performance tests.
- Also, in case you want to generate html reports for publication on the project site generated by Maven, execute the XSLT transformation from xml to html provided by Jmeter using the xml-maven-plugin:
<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>xml-maven-plugin</artifactId> <version>1.0-beta-2</version> <executions> <execution> <phase>pre-site</phase> <goals> <goal>transform</goal> </goals> </execution> </executions> <configuration> <transformationSets> <transformationSet> <dir>${project.build.directory}/jmeter-reports</dir> <stylesheet>src/test/resources/jmeter-results-detail-report_21.xsl</stylesheet> <outputDir>${project.build.directory}/site/jmeter-results</outputDir> <fileMappers> <fileMapper implementation="org.codehaus.plexus.components.io.filemappers.FileExtensionMapper"> <targetExtension>html</targetExtension> </fileMapper> </fileMappers> </transformationSet> </transformationSets> </configuration> </plugin>
- Create some Jmeter tests. I created 2 Jmeter tests, PerfTest1 and PerfTest2, which are parametized so different hostnames and ports can be specified for the tests. The maven profile “env-dev-jetty” (which is the default used) defines the Jmeter parameters for Jetty in the “jmeterUserProperties” section. We have a separate profile to run Jmeter against against the application deployed in a weblogic container.
- Copy the jmeter.properties file to the “/src/test/jmeter” folder in your project and modify the properties to fit your needs. For example, I had to modify some “jmeter.save.saveservice” parameters to enable additional output and also some “log_level” to see a bit more of detail in the jmeter.log file.
- Execute mvn to run the tests.
mvn integration-test
Up to this point everything was ok apart from the fact that when running in maven, the Jmeter tests where hanging just after finishing correctly. After some investigation, I found that the reason was the exit logic implemented in the maven Jmeter plugin. The plugin exits when the number of threads drops to the number just before the Jmeter start call. This works in most cases, but when you are also running an embedded Jetty server, the threads spawned to service the requests triggered by the Jmeter tests are counted as well, causing the wait. The plugin eventually exits when all these connection threads are closed after timing out.
The solution was to change the exit logic, monitoring the jmeter.log file as described here, instead of monitoring the number of threads. This should work in most cases.
- Integrate into Hudson, the continuous integration that we use. For that I installed the Jmeter Hudson plugin and configured it as shown below:
Hudson Jmeter Plugin Configuration
- In order to keep the Jmeter detailed performance test results, I configured Hudson to archive the relevant reports as build artifacts.
- The Hudson and Maven Jmeter plugins have different naming convention for the report files. The maven Jmeter plugin generates the report filename with a date which is not compatible with the static file name required by the Hudson Jmeter Plugin. Solution: I modified the maven plugin source code to generate a results file without the date in it (e.g. from PerfTest1-090413.xml to PerfTest1-result.xml). There is no disadvantage as Hudson will store the results for each build as an artifact, keeping a log.
- The Hudson plugin doesn’t really support collecting information about multiple Jmeter Tests at this moment. The way to go would be to group different performance tests into a single Jmeter file and create separate Hudson jobs for them.
- After all this setup has been done, you can execute some builds and the check that the Jmeter execution graphs works as they should. I created an example, containing errors and different response times to make it pretty. The graphs show the trend for only one of the Jmeter tests.
Hudson Jmeter Test Execution Trend
- Note the section “Last Successful Artifacts” that contains the html reports created during the maven site creation and archived for future reference. An example of this detail report is:
Jmeter Detailed report as Hudson Artifact
on May 7th, 2009 at 9:13 am
I am using ant script to generate Jmeter Report with Hudson Tool. i uploaded the jmeter plugin .
Build script generating the Jmeter Report file . But the Graph for the Performance Test is not appearing in Hudson
Please Help me.
Thank you in Advance
on May 8th, 2009 at 10:15 am
Hi Rathina,
I understand that you have everything running up to the generation of jmeter reports in Hudson, but the jmeter Hudson plugin is not picking up the reports.
That normally happens when the path to the Jmeter report is not correctly introduced in the configuration item (see Hudson Jmeter Plugin Configuration picture above).
Make sure the path is correct (remember that is relative to the project workspace) and check the output at the end of the build. If there is any problem accessing the reports you should see some error messages.
Hope this helps,
Jacobo
on May 8th, 2009 at 12:29 pm
Hi guys,
I had a similar problem today. The trick is to not create a “Build a maven2 project” but a “Build a free-style software project”. Took me a while to find out but it should work.
Cheers,
Martin
on May 20th, 2009 at 5:04 pm
Hi,
I had to solve the same problem on my hudson installation. I’ve figured out that trend reports/graphs for jmeter won’t appear unless your build is marked as SUCCESS.
You can check it in plugin’s code : http://fisheye4.atlassian.com/browse/hudson/tags/jmeter-0.2/src/main/java/hudson/plugins/jmeter/JMeterProjectAction.java?r=15892
check lines 173 and 207. Graphs are filled with data only when this condition is true :
Result.SUCCESS.equals(currentBuild.getResult()).
Make your build stable for at least two builds (by disabling some thresholds or tests) and you will see the graphs. One stable build is not enough - there will be only one point to plot on the graph :). Of course this is not a final solution.
It is possible to setup load tests as a separate job in Hudson (job that will run after main build). There are no problems with failed or unstable builds then.
Tom
on Jun 16th, 2009 at 9:02 am
I have a free style project where I am using windows batch command to run jmeter test plan. Everything works expect the JMeter trend report. I am not sure what I am doing wrong.
REM — launch JMeter project
cmd.exe /C jmeter.bat -n -t”C:\testplan.jmx” -l “C:\JmeterResults\%BUILD_NUMBER%\Results.jtl”
REM —- Dump Results.jtl to a folder inside workspace for graphing
xcopy “C:\JmeterResults\%BUILD_NUMBER%\Results.jtl” %WORKSPACE%\JMeterResults\ /Y
In the ‘JMeter Report’ section in post-build Actions, I have
JMeterResults/Results.jtl
(JMeterResults is a folder inside my workspace folder)
Any reason why I can’t view my JMeter Trend charts.
Thanks
Ali
on Jul 1st, 2009 at 11:41 pm
Ali, Make sure you don’t have any fail builds, if they already exists try remove them.
on Nov 9th, 2009 at 10:26 am
Hi,
I’m trying to use JMeter with Maven2 but i have an error. i think it is linked with this part:
“Copy the jmeter.properties file to the “/src/test/jmeter” folder in your project and modify the properties to fit your needs. For example, I had to modify some “jmeter.save.saveservice” parameters to enable additional output and also some “log_level” to see a bit more of detail in the jmeter.log file.”
Here is a sample of jmeter.log
2009/11/09 10:15:14 ERROR - jmeter.save.SaveService: Conversion error com.thoughtworks.xstream.converters.ConversionException: org/apache/commons/logging/LogFactory : org/apache/commons/logging/LogFactory
—- Debugging information —-
message : org/apache/commons/logging/LogFactory
cause-exception : java.lang.NoClassDefFoundError
cause-message : org/apache/commons/logging/LogFactory
first-jmeter-class : org.apache.jmeter.protocol.http.sampler.HTTPSampler2.(HTTPSampler2.java:204)
class : org.apache.jmeter.save.ScriptWrapper
required-type : org.apache.jmeter.protocol.http.sampler.SoapSampler
path : /jmeterTestPlan/hashTree/hashTree/hashTree/SoapSampler
line number : 29
——————————-
2009/11/09 10:15:14 ERROR - jmeter.JMeter: java.lang.NullPointerException
does any one have an idea?
Regards,
Dimitri
on Nov 10th, 2009 at 4:23 pm
i have put commons-logging and logkit into maven lib folder ans it works.
Regards,
Dimitri
on Nov 13th, 2009 at 10:55 pm
Great summary!
Folks may be interested in an alternative plugin called maven-chronos-plugin. It is able to run JMeter and doesn’t suffer from the problems caused by the approach JMeterMavenPlugin takes to running JMeter that you had to work around. Also it produces some really nice reports and graphs out-of-the-box:
http://mojo.codehaus.org/chronos-maven-plugin/index.html
The only downside is it requires an external install of JMeter. But I have posted instructions on how to use maven to get around this requirement:
http://gabenell.blogspot.com/2009/11/using-maven-chronos-without-external.html
on Jan 19th, 2010 at 12:56 am
I’m having the same issue as Dimitri. I made sure commons-logging etc were in my local repository and they are, but I’m still getting the same error in jmeter.log:
ERROR - jmeter.save.SaveService: Conversion error com.thoughtworks.xstream.converters.ConversionException: org/apache/commons/logging/LogFactory
: org/apache/commons/logging/LogFactory
—- Debugging information —-
message : org/apache/commons/logging/LogFactory
cause-exception : java.lang.NoClassDefFoundError
cause-message : org/apache/commons/logging/LogFactory
first-jmeter-class : org.apache.jmeter.protocol.http.control.CookieManager.setCookiePolicy(CookieManager.java:100)
class : org.apache.jmeter.save.ScriptWrapper
required-type : org.apache.jmeter.protocol.http.control.CookieManager
path : /jmeterTestPlan/hashTree/hashTree/hashTree/CookieManager
line number : 505
——————————-
Any other ideas?
Thanks,
Jared
on Feb 9th, 2010 at 4:02 pm
Gabe,
Could you please help me? I have ready jmeter report: aggregate_jmeter.jtl and I want to build graphs with maven-chronos-plugin. What I need to do?