Performance Tests with Jmeter, Maven and Hudson

Continuing with the series of blog posts regarding testing, automation and continuous integration, this time I will talk about how to integrate performance tests, in this case using Jmeter, with Maven and the Hudson continuous integration system. Jmeter is one of the main tools we use in our projects to create relevant performance tests, and automation and integration in our CI systems is essential. We also use SoapUI and Grinder depending on the platform, but we will cover those in future posts.

To integrate Jmeter and Maven, you must use the Maven Jmeter plugin, the first version of which is officially hosted here. There are are several posts discussing the use of this plugin that I have used as reference, particularly this one from James Lorenzen.

An updated version of this original plugin was released in google code by Ronald Alleva. It adds support for the latest version of Jmeter (2.3.2) and contains some additional enhacements such as parameterisation support. I decided to use this version to get this extra funcionality.

To make this post as simple and easy to follow as possible, my idea was to test Jmeter tests automation and integration against an application web running in an embeded Jetty container.

My objectives were:

  • Be able to run my Jmeter tests in a specific maven phase.
  • Create the relevant Jmeter test results html reports and publish them during the maven site phase.
  • Integrate Jmeter execution and reporting with Hudson, via the Hudson Jmeter plugin available here.

This post is a combination of James and Ron’s posts, adding solutions to problems I discovered while creating and executing my tests and showing the integration with Hudson.

Ronald describes pretty well here the different steps required in order to make the plugin work. I will not include these steps but rather comment on them.

  • Download the support jars and the maven Jmeter plugin and install them.
  • Add the dependency to your maven project. In our case, we want to execute our Jmeter tests when we start the embedded jetty container. We usually start/stop Jetty as part of the pre-integration and post-integration maven phases. We just need to add the Jmeter plugin execution to the integration-test phase (I have also included the jetty part below):
<profiles>
   <profile>
       <id>env-dev-jetty</id>
	<activation>
	   <activeByDefault>true</activeByDefault>
	</activation>
	<build>
	   <plugins>
		<plugin>
			<groupId>org.apache.jmeter</groupId>
			<artifactId>maven-jmeter-plugin</artifactId>
			<version>1.0</version>
			<executions>
				<execution>
					<id>jmeter-tests</id>
					<phase>integration-test</phase>
					<goals>
						<goal>jmeter</goal>
					</goals>
				</execution>
			</executions>
			<configuration>
				<reportDir>
					${project.build.directory}/jmeter-reports
				</reportDir>
				<jmeterUserProperties>
					   <hostname>localhost</hostname>
			       	  	   <port>9080</port>
       	  	  	  	</jmeterUserProperties>
			</configuration>
		</plugin>
		<plugin>
			<groupId>org.mortbay.jetty</groupId>
			<artifactId>maven-jetty-plugin</artifactId>
			<version>6.1.12rc1</version>
			<configuration>
				<webApp>
					${basedir}/target/${project.build.finalName}/${project.parent.name}-webapp-${project.version}.war
				</webApp>
				<contextPath>
					${project.parent.name}-webapp
				</contextPath>
				<connectors>
					<connector	implementation="org.mortbay.jetty.nio.SelectChannelConnector">
					   <port>9080</port>
					   <maxIdleTime>60000</maxIdleTime>
				        </connector>
				</connectors>
				<systemProperties>
					<systemProperty>
						<name>
						   pmtool.application.environment
						</name>
						<value>dev</value>
					</systemProperty>
				</systemProperties>
				<stopPort>9966</stopPort>
				<stopKey>foo</stopKey>
				<daemon>true</daemon>
				<scanIntervalSeconds>0</scanIntervalSeconds>
			</configuration>
			<executions>
				<execution>
					<id>start-jetty</id>
					<phase>pre-integration-test</phase>
					<goals>
						<goal>run-war</goal>
					</goals>
				</execution>
				<execution>
					<id>stop-jetty</id>
					<phase>post-integration-test</phase>
					<goals>
						<goal>stop</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
	</plugins>
   </build>
</profile>
</profiles>

In a real scenario where, as part of your project cycle, the application is deployed to a development or pre-production environment, we could use the verify phase to execute the performance tests.

  • Also, in case you want to generate html reports for publication on the project site generated by Maven, execute the XSLT transformation from xml to html provided by Jmeter using the xml-maven-plugin:
<plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>xml-maven-plugin</artifactId>
   <version>1.0-beta-2</version>
   <executions>
      <execution>
         <phase>pre-site</phase>
         <goals>
             <goal>transform</goal>
         </goals>
         </execution>
    </executions>
    <configuration>
         <transformationSets>
             <transformationSet>
                  <dir>${project.build.directory}/jmeter-reports</dir>
                  <stylesheet>src/test/resources/jmeter-results-detail-report_21.xsl</stylesheet>
                  <outputDir>${project.build.directory}/site/jmeter-results</outputDir>
                  <fileMappers>
                      <fileMapper implementation="org.codehaus.plexus.components.io.filemappers.FileExtensionMapper">
                         <targetExtension>html</targetExtension>
                      </fileMapper>
                  </fileMappers>
             </transformationSet>
         </transformationSets>
    </configuration>
</plugin>
  • Create some Jmeter tests. I created 2 Jmeter tests, PerfTest1 and PerfTest2, which are parametized so different hostnames and ports can be specified for the tests. The maven profile “env-dev-jetty” (which is the default used) defines the Jmeter parameters for Jetty in the “jmeterUserProperties” section. We have a separate profile to run Jmeter against against the application deployed in a weblogic container.
  • Copy the jmeter.properties file to the “/src/test/jmeter” folder in your project and modify the properties to fit your needs. For example, I had to modify some “jmeter.save.saveservice” parameters to enable additional output and also some “log_level” to see a bit more of detail in the jmeter.log file.
  • Execute mvn to run the tests.
mvn integration-test

Up to this point everything was ok apart from the fact that when running in maven, the Jmeter tests where hanging just after finishing correctly. After some investigation, I found that the reason was the exit logic implemented in the maven Jmeter plugin. The plugin exits when the number of threads drops to the number just before the Jmeter start call. This works in most cases, but when you are also running an embedded Jetty server, the threads spawned to service the requests triggered by the Jmeter tests are counted as well, causing the wait. The plugin eventually exits when all these connection threads are closed after timing out.

The solution was to change the exit logic, monitoring the jmeter.log file as described here, instead of monitoring the number of threads. This should work in most cases.

  • Integrate into Hudson, the continuous integration that we use. For that I installed the Jmeter Hudson plugin and configured it as shown below:
Hudson Jmeter Plugin Configuration

Hudson Jmeter Plugin Configuration

  • In order to keep the Jmeter detailed performance test results, I configured Hudson to archive the relevant reports as build artifacts.
  • The Hudson and Maven Jmeter plugins have different naming convention for the report files. The maven Jmeter plugin generates the report filename with a date which is not compatible with the static file name required by the Hudson Jmeter Plugin. Solution: I modified the maven plugin source code to generate a results file without the date in it (e.g. from PerfTest1-090413.xml to PerfTest1-result.xml). There is no disadvantage as Hudson will store the results for each build as an artifact, keeping a log.
  • The Hudson plugin doesn’t really support collecting information about multiple Jmeter Tests at this moment. The way to go would be to group different performance tests into a single Jmeter file and create separate Hudson jobs for them.
  • After all this setup has been done, you can execute some builds and the check that the Jmeter execution graphs works as they should. I created an example, containing errors and different response times to make it pretty. The graphs show the trend for only one of the Jmeter tests.
Hudson Jmeter Test Execution Trend

Hudson Jmeter Test Execution Trend

  • Note the section “Last Successful Artifacts” that contains the html reports created during the maven site creation and archived for future reference. An example of this detail report is:
Jmeter Detailed report as Hudson Artifact

Jmeter Detailed report as Hudson Artifact

18 Comments on “Performance Tests with Jmeter, Maven and Hudson”

  1. #1 Rathina Sabapathy
    on May 7th, 2009 at 9:13 am

    I am using ant script to generate Jmeter Report with Hudson Tool. i uploaded the jmeter plugin .
    Build script generating the Jmeter Report file . But the Graph for the Performance Test is not appearing in Hudson

    Please Help me.

    Thank you in Advance

  2. #2 Jacobo Matute
    on May 8th, 2009 at 10:15 am

    Hi Rathina,

    I understand that you have everything running up to the generation of jmeter reports in Hudson, but the jmeter Hudson plugin is not picking up the reports.
    That normally happens when the path to the Jmeter report is not correctly introduced in the configuration item (see Hudson Jmeter Plugin Configuration picture above).
    Make sure the path is correct (remember that is relative to the project workspace) and check the output at the end of the build. If there is any problem accessing the reports you should see some error messages.

    Hope this helps,
    Jacobo

  3. #3 mbreest
    on May 8th, 2009 at 12:29 pm

    Hi guys,

    I had a similar problem today. The trick is to not create a “Build a maven2 project” but a “Build a free-style software project”. Took me a while to find out but it should work.

    Cheers,
    Martin

  4. #4 Tom Poręba
    on May 20th, 2009 at 5:04 pm

    Hi,
    I had to solve the same problem on my hudson installation. I’ve figured out that trend reports/graphs for jmeter won’t appear unless your build is marked as SUCCESS.

    You can check it in plugin’s code : http://fisheye4.atlassian.com/browse/hudson/tags/jmeter-0.2/src/main/java/hudson/plugins/jmeter/JMeterProjectAction.java?r=15892

    check lines 173 and 207. Graphs are filled with data only when this condition is true :

    Result.SUCCESS.equals(currentBuild.getResult()).

    Make your build stable for at least two builds (by disabling some thresholds or tests) and you will see the graphs. One stable build is not enough – there will be only one point to plot on the graph :) . Of course this is not a final solution.

    It is possible to setup load tests as a separate job in Hudson (job that will run after main build). There are no problems with failed or unstable builds then.

    Tom

  5. #5 Ali Raza
    on Jun 16th, 2009 at 9:02 am

    I have a free style project where I am using windows batch command to run jmeter test plan. Everything works expect the JMeter trend report. I am not sure what I am doing wrong.

    REM — launch JMeter project
    cmd.exe /C jmeter.bat -n -t”C:\testplan.jmx” -l “C:\JmeterResults\%BUILD_NUMBER%\Results.jtl”

    REM —- Dump Results.jtl to a folder inside workspace for graphing
    xcopy “C:\JmeterResults\%BUILD_NUMBER%\Results.jtl” %WORKSPACE%\JMeterResults\ /Y

    In the ‘JMeter Report’ section in post-build Actions, I have
    JMeterResults/Results.jtl

    (JMeterResults is a folder inside my workspace folder)

    Any reason why I can’t view my JMeter Trend charts.

    Thanks

    Ali

  6. #6 Ken Huang
    on Jul 1st, 2009 at 11:41 pm

    Ali, Make sure you don’t have any fail builds, if they already exists try remove them.

  7. #7 Dimitri
    on Nov 9th, 2009 at 10:26 am

    Hi,
    I’m trying to use JMeter with Maven2 but i have an error. i think it is linked with this part:
    “Copy the jmeter.properties file to the “/src/test/jmeter” folder in your project and modify the properties to fit your needs. For example, I had to modify some “jmeter.save.saveservice” parameters to enable additional output and also some “log_level” to see a bit more of detail in the jmeter.log file.”

    Here is a sample of jmeter.log
    2009/11/09 10:15:14 ERROR – jmeter.save.SaveService: Conversion error com.thoughtworks.xstream.converters.ConversionException: org/apache/commons/logging/LogFactory : org/apache/commons/logging/LogFactory
    —- Debugging information —-
    message : org/apache/commons/logging/LogFactory
    cause-exception : java.lang.NoClassDefFoundError
    cause-message : org/apache/commons/logging/LogFactory
    first-jmeter-class : org.apache.jmeter.protocol.http.sampler.HTTPSampler2.(HTTPSampler2.java:204)
    class : org.apache.jmeter.save.ScriptWrapper
    required-type : org.apache.jmeter.protocol.http.sampler.SoapSampler
    path : /jmeterTestPlan/hashTree/hashTree/hashTree/SoapSampler
    line number : 29
    ——————————-
    2009/11/09 10:15:14 ERROR – jmeter.JMeter: java.lang.NullPointerException

    does any one have an idea?

    Regards,
    Dimitri

  8. #8 Dimitri
    on Nov 10th, 2009 at 4:23 pm

    i have put commons-logging and logkit into maven lib folder ans it works.

    Regards,
    Dimitri

  9. #9 Gabe
    on Nov 13th, 2009 at 10:55 pm

    Great summary!

    Folks may be interested in an alternative plugin called maven-chronos-plugin. It is able to run JMeter and doesn’t suffer from the problems caused by the approach JMeterMavenPlugin takes to running JMeter that you had to work around. Also it produces some really nice reports and graphs out-of-the-box:

    http://mojo.codehaus.org/chronos-maven-plugin/index.html

    The only downside is it requires an external install of JMeter. But I have posted instructions on how to use maven to get around this requirement:

    http://gabenell.blogspot.com/2009/11/using-maven-chronos-without-external.html

  10. #10 jared
    on Jan 19th, 2010 at 12:56 am

    I’m having the same issue as Dimitri. I made sure commons-logging etc were in my local repository and they are, but I’m still getting the same error in jmeter.log:

    ERROR – jmeter.save.SaveService: Conversion error com.thoughtworks.xstream.converters.ConversionException: org/apache/commons/logging/LogFactory
    : org/apache/commons/logging/LogFactory

    —- Debugging information —-
    message : org/apache/commons/logging/LogFactory
    cause-exception : java.lang.NoClassDefFoundError
    cause-message : org/apache/commons/logging/LogFactory
    first-jmeter-class : org.apache.jmeter.protocol.http.control.CookieManager.setCookiePolicy(CookieManager.java:100)
    class : org.apache.jmeter.save.ScriptWrapper
    required-type : org.apache.jmeter.protocol.http.control.CookieManager
    path : /jmeterTestPlan/hashTree/hashTree/hashTree/CookieManager
    line number : 505
    ——————————-

    Any other ideas?

    Thanks,
    Jared

  11. #11 iroller
    on Feb 9th, 2010 at 4:02 pm

    Gabe,
    Could you please help me? I have ready jmeter report: aggregate_jmeter.jtl and I want to build graphs with maven-chronos-plugin. What I need to do?

  12. #12 Tarun K
    on Sep 6th, 2010 at 5:09 am

    Even I had problem with having reports being shown in Hudson as there is date stamp attached to report. I have tried following to get through while specifying path to JMeter reports -

    1. Specifying full path –
    C:/SelNG/jmeter2/target/jmeter-reports/GoogleAdvanceSearch-100906.xml

    This works but the problem is time stamp attached to report -
    “GoogleAdvanceSearch-100906.xml” which would change with respect to date, hence each time I would have to specify new path.

    2. Using regular expression -
    **/*.xml
    This is suggested to be used here – http://wiki.hudson-ci.org/display/HUDSON/Performance+Plugin
    But it does not work and I encounter following exception -

    ************************
    Performance: Recording JMeter reports ‘**/*.xml’
    Performance: no JMeter files matching ‘**/*.xml’ have been found. Has the report generated?. Setting Build to FAILURE
    Finished: FAILURE
    ************************

    Do u have any idea why regular expression does not work for me?

    Thanks in advance
    ~ T

  13. #13 kviswanathan
    on Oct 7th, 2010 at 1:45 pm

    Hi all,

    I am using Performance plugin to display the *.jtl results. The result gets displayed in report form but the trend graph for the .jtl files are not displaying anything.
    I have enabled cookies in the browser.

    Hudson Version:1.369

    Please help!

    TIA

  14. #14 Debjyoti
    on Oct 26th, 2010 at 12:49 pm

    Hi myself debjyoti biswas , Software Tester at Spatial ideas .Recently I have download Apache j meter for load test .I am able to run the JMeter , but my challenge is to understand the output result of the executed programs . So can any body help me out to understand the Apache jmeter.
    Not able to under stand the out put graph result also.
    Anybody knowns any online tutorial to understand the
    Apache JMeter results.
    As I Am very much new in Load Testing Please Help me.
    THanks ,
    Debjyoti

  15. #15 Viktor
    on Jan 21st, 2012 at 12:37 am

    Hi,
    I am getting PermGen exceptions as the following. It gets OutOfMemory exceptions when JMeter plugin start running tests from jmx files. By any chance, do you know how I can fix it ? Where the memory should be increased ?

    java.lang.OutOfMemoryError: PermGen space

    Thanks.

  16. #16 How to run JMeter tests with Maven | Ribble's Code
    on Apr 16th, 2012 at 11:58 pm

    [...] a lot of help from some other people’s posts on this subject.  Jacobo Matute’s post Performance Tests with Jmeter, Maven and Hudson was a big help, but fortunately slightly out of [...]

  17. #17 Dzmitry Kashlach
    on Feb 8th, 2013 at 10:52 am

    Hi,

    Thank you for the guide. In addition, I can suggest another solution, how to integrate JMeter tests in to building process:
    – install Jenkins/(ex-Hudson);
    – download Jenkins plugin
    – register to performance testing cloud and upload JMeter script
    – start build – your test will be started automatically in the cloud.

    The main benifit of using this cloud is that it is possible to configure test with up to 80000 users in 5-10 minutes.

    P.S. While I’m working for Blazemeter, it’s not a marketing comment ;-)

  18. #18 Dzmitry Kashlach
    on Feb 8th, 2013 at 12:33 pm

    Another way to set JMeter with Continuous Integration is to use JMeter testing cloud with Jenkins plugin.
    This will allow to see more comprehensive reports, than using Jmeter out-of-box facilities.

Leave a Comment