Thursday, January 14, 2010

Run your junit tests concurrently with maven, junit (and maybe also spring3) in 5 minutes or less!

Surefire 2.5 is released, and it contains the concurrent junit patches. In this post I'll give the quick rundown of how to try out your current maven based build in a concurrent fashion. Just a few initial thoughts:

Will my tests run concurrently ?


Probably not, as is. Most existing test fixtures that people use have singletons and shared state that needs to be fixed first. This may be everything from a shared file, shared TCP/IP port or a static member variable in some base class that can't be static any more.

It took me about a day to fix these things in my current project. My project is large and complex. using all sorts of dark spring magics. Your mileage may vary.

What performance gain can I expect ?


For an IO-bound test (integration test/selenium test etc), the sky's the limit ;)

For a fairly optimized unit-test set, expect little or no gain - maybe 15-20%.
The reason for this is that until jdk7, all classloading and all classpath bound resource access is synchronized across the VM. Running unit tests is mostly about classloading and classpath resources :( If your unit-tests have other kinds of I/O bindings you may be luckier.

If you're using spring inside your test fixtures (like we do), you can probably squeeze a few drops of speed out of it by making lazy-init==true globally. You're going to test it all anyway, right ?

How to do it!


Make sure you upgrade to surefire-2.5:


<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.5</version>
</plugin>


Make sure you upgrade to junit 4.8.1 (still not in maven central repo, install locally):

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>


(You /can/ use 4.7 but it still has a couple of concurrency bugs in it that were fixed in 4.8.1)

Fix your surefire configuration:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
....
<parallel>classes</parallel>
</configuration>
</plugin>


Legal values for parallel are "classes, methods or both". Now you can run, but you should probably read the rest of this post first...

Classes, methods or both ?


From a concurrency perspective, "classes" is probably the easiest to start with, since it will probably run into the smallest number of troubles in your test fixtures. I really recommend you try to get both working, since you'll probably need to identify any additional issues raised by using "methods". You want a test-suite you can trust, right ?

The fine print


One drawback of the current default concurrency implementation in junit is that it does not allow you to constrain threads, which is desirable for almost all use cases I am aware of. But junit is extendable, and I have made a supplemental add-on that allows threads to be configured too. Surefire knows to invoke this one if it's present on your classpath, meaning you'll get additional options available:

The configurable-parallel-computer still has a few issues that are unsolved, so you
may consider dropping by the issue tracker to check if these will bother you before adding it to your project

git clone http://github.com/krosenvold/configurable-parallel-computer.git
cd configurable-parallel-computer
mvn install


<dependency>
<groupId>org.jdogma.junit</groupId>
<artifactId>configurable-parallel-computer</artifactId>
<version>1.5</version>
<scope>test</scope>
</dependency>




<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
....
<parallel>classes</parallel>
<threadCount>2</threadCount>
</configuration>
</plugin>


You can also use the setting "perCoreThreadCount", which scales threadcount per CPU core.

The configurable-parallel-computer project also contains a better output demultiplexer than the one in surefire (which has a ketchup-bottle behaviour). Surefire knows about this one too, so adding configurable-parallel-computer to your classpath will give you smooth project output. I'm sure we'll get the improved version into surefire at some later time.


Spring


If you're using spring you need to use a forked version of spring-test. The patch I submitted to spring is targeted for 3.1, whatever that may mean in practice:

git clone http://github.com/krosenvold/org.springframework.test.git
cd org.springframework.test
mvn install

You need to replace your dependency on "spring-test" to the forked version:


<dependency>
<groupId>org.rosenvold.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>3.0.0.RELEASE</version>
<scope>test</scope>
</dependency>

Remember to also suppress the transitively dependent spring-test artifacts, like this:

<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
<version>3.0.0.RELEASE</version>
<exclusions>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
</exclusion>
</exclusions>
</dependency>


Make sure mvn dependency:tree does not contain the spring version of spring-test. Any inclusion of the 3.0 artifact WILL get you into trouble.

If you're using a Mock Session Scope based on the Spring Session Scope (with a custom ContextLoader), you need to make sure you override the synchronized methods in the base class with your own implementations, or your tests may deadlock on the session scope. This is a real bug in spring, but one that probably does not occur to often in real-life scenarios; but quite a lot when running concurrent tests.

Off you go !

9 comments:

  1. Very interesting post. I have been searching it for some time.

    Will try it soon.

    Maralc

    ReplyDelete
  2. Finally all there, I have been waiting for your code to be introduced in surefire and junit quite long. Great work.
    Did you think about publishing the configurable-parallel-computer to any maven repository?

    ReplyDelete
  3. Thanks for this awesome work.

    Do you have a hint for me: how do you run your tests continuos all the time, without the necessity to push run as junit tests all the time and get a notification if your tests fail?

    Here are two links, which will explain, what I'm searching for:

    http://www.zenspider.com/ZSS/Products/ZenTest/
    http://vimeo.com/2680374

    Switching from Ruby to Java again makes me miss some nice features :)

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Could you elaborate how surefire's "parallel" works internally? No matter what I try, I always get:
    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

    Could it be that "parallel" doesn't work with custom @RunWith runners, but only with, say, subclasses of org.junit.runners.ParentRunner?

    ReplyDelete
    Replies
    1. I know this was a little while a go, and its a bit of a long shot - but I think I'm seeing a similar issue, did you ever find out what was causing this?

      Thanks for any thoughts,

      Matt

      Delete
  6. This is fantastic work, thanks for all the work on this and the article.

    Roll on Spring 3.1!

    ReplyDelete
  7. As of surefire 2.7, configurable-parallel computer is part of surefire and no longer needed.

    ReplyDelete
  8. "For a fairly optimized unit-test set, expect little or no gain - maybe 15-20%."

    I'm working on a new test runner which will implement class loader caching of the library dependencies. Initial benchmarks indicate that class loader caching will speed up the suite run times considerably. In this project (only unit tests) the suite run time on 4 threads decreased from 3.7 seconds to 1.4 seconds: http://blog.orfjackal.net/2013/02/faster-junit-tests-with-jumi-test.html

    These benchmarks were run on JDK 7, so class loader caching bings considerable benefits even over JDK 7's parallel capable class loaders. Also the JIT compiler starts kicking in after running the tests a couple of times.

    ReplyDelete