Thursday, January 14, 2010

Run your junit tests concurrently with maven, junit (and maybe also spring3) in 5 minutes or less!

Surefire 2.5 is released, and it contains the concurrent junit patches. In this post I'll give the quick rundown of how to try out your current maven based build in a concurrent fashion. Just a few initial thoughts:

Will my tests run concurrently ?


Probably not, as is. Most existing test fixtures that people use have singletons and shared state that needs to be fixed first. This may be everything from a shared file, shared TCP/IP port or a static member variable in some base class that can't be static any more.

It took me about a day to fix these things in my current project. My project is large and complex. using all sorts of dark spring magics. Your mileage may vary.

What performance gain can I expect ?


For an IO-bound test (integration test/selenium test etc), the sky's the limit ;)

For a fairly optimized unit-test set, expect little or no gain - maybe 15-20%.
The reason for this is that until jdk7, all classloading and all classpath bound resource access is synchronized across the VM. Running unit tests is mostly about classloading and classpath resources :( If your unit-tests have other kinds of I/O bindings you may be luckier.

If you're using spring inside your test fixtures (like we do), you can probably squeeze a few drops of speed out of it by making lazy-init==true globally. You're going to test it all anyway, right ?

How to do it!


Make sure you upgrade to surefire-2.5:


<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.5</version>
</plugin>


Make sure you upgrade to junit 4.8.1 (still not in maven central repo, install locally):

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>


(You /can/ use 4.7 but it still has a couple of concurrency bugs in it that were fixed in 4.8.1)

Fix your surefire configuration:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
....
<parallel>classes</parallel>
</configuration>
</plugin>


Legal values for parallel are "classes, methods or both". Now you can run, but you should probably read the rest of this post first...

Classes, methods or both ?


From a concurrency perspective, "classes" is probably the easiest to start with, since it will probably run into the smallest number of troubles in your test fixtures. I really recommend you try to get both working, since you'll probably need to identify any additional issues raised by using "methods". You want a test-suite you can trust, right ?

The fine print


One drawback of the current default concurrency implementation in junit is that it does not allow you to constrain threads, which is desirable for almost all use cases I am aware of. But junit is extendable, and I have made a supplemental add-on that allows threads to be configured too. Surefire knows to invoke this one if it's present on your classpath, meaning you'll get additional options available:

The configurable-parallel-computer still has a few issues that are unsolved, so you
may consider dropping by the issue tracker to check if these will bother you before adding it to your project

git clone http://github.com/krosenvold/configurable-parallel-computer.git
cd configurable-parallel-computer
mvn install


<dependency>
<groupId>org.jdogma.junit</groupId>
<artifactId>configurable-parallel-computer</artifactId>
<version>1.5</version>
<scope>test</scope>
</dependency>




<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
....
<parallel>classes</parallel>
<threadCount>2</threadCount>
</configuration>
</plugin>


You can also use the setting "perCoreThreadCount", which scales threadcount per CPU core.

The configurable-parallel-computer project also contains a better output demultiplexer than the one in surefire (which has a ketchup-bottle behaviour). Surefire knows about this one too, so adding configurable-parallel-computer to your classpath will give you smooth project output. I'm sure we'll get the improved version into surefire at some later time.


Spring


If you're using spring you need to use a forked version of spring-test. The patch I submitted to spring is targeted for 3.1, whatever that may mean in practice:

git clone http://github.com/krosenvold/org.springframework.test.git
cd org.springframework.test
mvn install

You need to replace your dependency on "spring-test" to the forked version:


<dependency>
<groupId>org.rosenvold.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>3.0.0.RELEASE</version>
<scope>test</scope>
</dependency>

Remember to also suppress the transitively dependent spring-test artifacts, like this:

<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
<version>3.0.0.RELEASE</version>
<exclusions>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
</exclusion>
</exclusions>
</dependency>


Make sure mvn dependency:tree does not contain the spring version of spring-test. Any inclusion of the 3.0 artifact WILL get you into trouble.

If you're using a Mock Session Scope based on the Spring Session Scope (with a custom ContextLoader), you need to make sure you override the synchronized methods in the base class with your own implementations, or your tests may deadlock on the session scope. This is a real bug in spring, but one that probably does not occur to often in real-life scenarios; but quite a lot when running concurrent tests.

Off you go !

17 comments:

  1. Very interesting post. I have been searching it for some time.

    Will try it soon.

    Maralc

    ReplyDelete
  2. Finally all there, I have been waiting for your code to be introduced in surefire and junit quite long. Great work.
    Did you think about publishing the configurable-parallel-computer to any maven repository?

    ReplyDelete
  3. Thanks for this awesome work.

    Do you have a hint for me: how do you run your tests continuos all the time, without the necessity to push run as junit tests all the time and get a notification if your tests fail?

    Here are two links, which will explain, what I'm searching for:

    http://www.zenspider.com/ZSS/Products/ZenTest/
    http://vimeo.com/2680374

    Switching from Ruby to Java again makes me miss some nice features :)

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Could you elaborate how surefire's "parallel" works internally? No matter what I try, I always get:
    Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

    Could it be that "parallel" doesn't work with custom @RunWith runners, but only with, say, subclasses of org.junit.runners.ParentRunner?

    ReplyDelete
    Replies
    1. I know this was a little while a go, and its a bit of a long shot - but I think I'm seeing a similar issue, did you ever find out what was causing this?

      Thanks for any thoughts,

      Matt

      Delete
  6. This is fantastic work, thanks for all the work on this and the article.

    Roll on Spring 3.1!

    ReplyDelete
  7. As of surefire 2.7, configurable-parallel computer is part of surefire and no longer needed.

    ReplyDelete
  8. "For a fairly optimized unit-test set, expect little or no gain - maybe 15-20%."

    I'm working on a new test runner which will implement class loader caching of the library dependencies. Initial benchmarks indicate that class loader caching will speed up the suite run times considerably. In this project (only unit tests) the suite run time on 4 threads decreased from 3.7 seconds to 1.4 seconds: http://blog.orfjackal.net/2013/02/faster-junit-tests-with-jumi-test.html

    These benchmarks were run on JDK 7, so class loader caching bings considerable benefits even over JDK 7's parallel capable class loaders. Also the JIT compiler starts kicking in after running the tests a couple of times.

    ReplyDelete
  9. It’s really great information Thanks for sharing. Best Manual Testing Training in Bangalore, BTM layout. My Class Training Academy training center for certified course, learning on Manual Testing Course by expert faculties, also provides job placement for fresher, experience job seekers.

    ReplyDelete
  10. Thanks for posting
    my website techwolk is related to technology

    ReplyDelete
  11. Excellent!  Your blog has quite a few readers. How did you get all of these people to look at your article I’m jealous! I’m still studying all about posting articles on the web. I’m going to click on more articles on your website to get a better understanding how to attract more people. Thanks!

    ReplyDelete
  12. Quite easily, the article is in reality the sweetest on this laudable topic. I agree with your conclusions and definitely will eagerly look forward to your approaching updates. Simply just saying thanks will certainly not simply just be sufficient, for the outstanding lucidity in your writing. I can right away grab your rss feed to stay privy of any updates. Good work and much success in your business efforts!

    ReplyDelete
  13. Hi! I understand this is kind of off-topic but I had to ask. Does managing a well-established blog like yours require a lot of work? I am completely new to operating a blog however I do write in my diary daily. I’d like to start a blog so I can share my personal experience and views online. Please let me know if you have any kind of suggestions or tips for new aspiring blog owners. Thankyou!

    What your stating is absolutely genuine. I know that everyone ought to say the identical factor, but I just feel that you set it in a way that absolutely everyone can realize. I also adore the photographs you set in here. They fit so nicely with what youre hoping to say. Im guaranteed youll attain so numerous people today with what youve got to say.

    These moles may possibly be irregular in size and color and that is what can make them this type of wellness danger. When you have been born with this particular problem you might also be more likely to develop Melanoma and so you might have to get the required precautions with regards to protecting your pores and skin and your well being.

    This is such a great post, and was thinking much the same myself. It’s certainly an opinion I agree with.

    Enjoyed this article. I believe that the writer took an rationale perspective and made some pivotale ideas.

    Aw, it was an extremely good post. In notion I must place in writing such as this moreover – spending time and actual effort to make a really good article… but so what can I say… I procrastinate alot and no means apparently go completed.

    You need to experience a tournament for just one of the most effective blogs on the web. I’ll recommend this page!

    Thank you for yet another great informative article, I’m a loyal reader to this blog and I can’t stress enough how much valuable information I’ve learned from reading your content. I really appreciate all the hard work you put into this great site.

    When I originally commented I clicked the -Notify me when new comments are added- checkbox and now when a comment is added I buy four emails concentrating on the same comment. Perhaps there is in any manner you can get rid of me from that service? Thanks!

    ReplyDelete
  14. Amazing Post, Thank you for sharing this post really this is awesome and very useful.

    ReplyDelete