Run your junit tests concurrently with maven, junit (and maybe also spring3) in 5 minutes or less!

Surefire 2.5 is released, and it contains the concurrent junit patches. In this post I'll give the quick rundown of how to try out your current maven based build in a concurrent fashion. Just a few initial thoughts:

Will my tests run concurrently ?

Probably not, as is. Most existing test fixtures that people use have singletons and shared state that needs to be fixed first. This may be everything from a shared file, shared TCP/IP port or a static member variable in some base class that can't be static any more.

It took me about a day to fix these things in my current project. My project is large and complex. using all sorts of dark spring magics. Your mileage may vary.

What performance gain can I expect ?

For an IO-bound test (integration test/selenium test etc), the sky's the limit ;)

For a fairly optimized unit-test set, expect little or no gain - maybe 15-20%.
The reason for this is that until jdk7, all classloading and all classpath bound resource access is synchronized across the VM. Running unit tests is mostly about classloading and classpath resources :( If your unit-tests have other kinds of I/O bindings you may be luckier.

If you're using spring inside your test fixtures (like we do), you can probably squeeze a few drops of speed out of it by making lazy-init==true globally. You're going to test it all anyway, right ?

How to do it!

Make sure you upgrade to surefire-2.5:

 <plugin>
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-surefire-plugin</artifactId>
     <version>2.5</version>
 </plugin>

Make sure you upgrade to junit 4.8.1 (still not in maven central repo, install locally):

 <dependency>
     <groupId>junit</groupId>
     <artifactId>junit</artifactId>
     <version>4.8.1</version>
     <scope>test</scope>
  </dependency>

(You /can/ use 4.7 but it still has a couple of concurrency bugs in it that were fixed in 4.8.1)

Fix your surefire configuration:

  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
          ....
          <parallel>classes</parallel>
      </configuration>
  </plugin>

Legal values for parallel are "classes, methods or both". Now you can run, but you should probably read the rest of this post first...

Classes, methods or both ?

From a concurrency perspective, "classes" is probably the easiest to start with, since it will probably run into the smallest number of troubles in your test fixtures. I really recommend you try to get both working, since you'll probably need to identify any additional issues raised by using "methods". You want a test-suite you can trust, right ?

The fine print

One drawback of the current default concurrency implementation in junit is that it does not allow you to constrain threads, which is desirable for almost all use cases I am aware of. But junit is extendable, and I have made a supplemental add-on that allows threads to be configured too. Surefire knows to invoke this one if it's present on your classpath, meaning you'll get additional options available:

The configurable-parallel-computer still has a few issues that are unsolved, so you
may consider dropping by the issue tracker to check if these will bother you before adding it to your project

git clone http://github.com/krosenvold/configurable-parallel-computer.git
cd configurable-parallel-computer
mvn install

  <dependency>
     <groupId>org.jdogma.junit</groupId>
     <artifactId>configurable-parallel-computer</artifactId>
     <version>1.5</version>
     <scope>test</scope>
  </dependency>

  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
          ....
          <parallel>classes</parallel>
          <threadCount>2</threadCount>
      </configuration>
  </plugin>

You can also use the setting "perCoreThreadCount", which scales threadcount per CPU core.

The configurable-parallel-computer project also contains a better output demultiplexer than the one in surefire (which has a ketchup-bottle behaviour). Surefire knows about this one too, so adding configurable-parallel-computer to your classpath will give you smooth project output. I'm sure we'll get the improved version into surefire at some later time.

Spring

If you're using spring you need to use a forked version of spring-test. The patch I submitted to spring is targeted for 3.1, whatever that may mean in practice:

git clone http://github.com/krosenvold/org.springframework.test.git
cd org.springframework.test
mvn install

You need to replace your dependency on "spring-test" to the forked version:

<dependency>
   <groupId>org.rosenvold.springframework</groupId>
   <artifactId>spring-test</artifactId>
   <version>3.0.0.RELEASE</version>
   <scope>test</scope>
</dependency>

Remember to also suppress the transitively dependent spring-test artifacts, like this:

<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-aspects</artifactId>
  <version>3.0.0.RELEASE</version>
  <exclusions>
     <exclusion>
          <groupId>org.springframework</groupId>
          <artifactId>spring-test</artifactId>
     </exclusion>
  </exclusions>
</dependency>

Make sure mvn dependency:tree does not contain the spring version of spring-test. Any inclusion of the 3.0 artifact WILL get you into trouble.

If you're using a Mock Session Scope based on the Spring Session Scope (with a custom ContextLoader), you need to make sure you override the synchronized methods in the base class with your own implementations, or your tests may deadlock on the session scope. This is a real bug in spring, but one that probably does not occur to often in real-life scenarios; but quite a lot when running concurrent tests.

Off you go !