My recent post about running unit tests in parallel has led to some interesting food for thought:
When running regular unit tests, running with 2 cores seems to be no faster than one core. 4 cores (and arguably a newer CPU architecture) had a significant impact, where performance nearly doubled. The hyperthreaded quad-core i7 920 was *slower* than a 3Ghz core 2 duo when running single threaded, but nearly twice as fast in a multithreaded scenario. What's happening ?
I looked into the issue with jprofiler, but there is no obvious lock contention or other clearly visible bottleneck. The only explanation I can think of so far is this:
The test I was running was extremely simplistic, and there was no real cpu requirement in running the test methods. So the overhead of creating thread pools and all that actually amounts to just as much computation as running the tests. Likely ?
After my summer vacation, I will try to find out. I probably need to run with more realistic test cases. I figured I'd build surefire with threads, because thats a fairly heavy build. In the mean time, your thoughts are welcome.
Sunday, July 5, 2009
Wednesday, July 1, 2009
Swimming in a sea of patches...
Note; Most of the content in this post has been surpassed by a newer entry.
If you're willing to apply a few patches, you can now run your junit tests in parallel. There are two main use-cases for this; one is to make your regular unit tests run faster, the other is for io-bound tests, where you want to run several in parallel because you're waiting for external systems.
Just to whet your appetite, here are some numbers from a testcase (running 1000 instances of a simple unit-test-class):
SC = SingleThreadedComputer (Computer, Junit47-snapshot class)
PC = ParallelComputer (ParallelComputer, Junit4.7-snapshot class)
CPC = ConfigurableParallelComputer (My project, see references)
C=Classes (one thread per class)
M=Methods (one thread per method)
|Core 2 duo 3.14Ghz||Intel i7 920|
|PC(Classes, Method, Unlimited threads)||406||806|
|CPC(Classes, Unlimited threads)||203||116|
|CPC(Classes,Methods, Unlimited threads)||266||147|
Running time in ms, 1000 simple tests
The table shows that for most dual-core solutions, running with parallel threads is currently never any faster than running non-threaded. Switching to the brand-new Intel i7 (4 hyperthreaded cores) has a marked effect, where simple unit test performance is doubled.
The lack of punch does, of course, change the moment your tests start doing IO (Integration tests, selenium tests and other web-tests).
The juicy stuff
svn co https://svn.openqa.org/svn/selenium-rc
Second build with mvn clean install: 102 seconds.
With 80 threads class-parallel on the i7: 37 seconds.
Class-parallel works easily on most tests.
A sea of patches
Getting all of this to work was not without its problems, though. To get this running, I had to patch just about half the libraries in use in my current stack. Furthermore, there's a huge difference between getting it to run and getting it to run without hiccups - every time. These patches insure that your infrastructure works as it should, but are your tests concurrent ?
See below for how to apply these patches in a simple manner.
Spring (3.0 trunk):
AspectJ: Patch trunk with this patch (if using spring) (This patch is included in version 1.6.6)
Several patches needed for thread safety:
Maven: You need a patched version of surefire that supports junit 4.7:
You'll also probably want to be using the ConfigurableParallelComputer instead of the ParallelComputer that's included with JUnit.
With these patches in place you're smooth sailing ;)
Applying the patches
Note: If you did this on the pre rc1 version of this patchset, I recommend that you start from a fresh copy of everything this time.
If you're using maven you could try this sequence:
git clone git://github.com/krosenvold/spring-framework-concurrent-junit-patch.git
git clone git://github.com/krosenvold/junit.git
git clone git://github.com/krosenvold/configurable-parallel-computer.git
git clone git://github.com/krosenvold/spring-test.git
git clone git://github.com/krosenvold/spring-concurrency-test.git
mvn install # Test project to verify that everything is ok
Your new versions
JUnit should be modified to 4.7-CONCURRENT
spring-test should have version number 3.0.0.RC1-CONCURRENT
Surefire should be modified to 2.5-SNAPSHOT
You must update your aspectj version to 1.6.6 in your project. Spring does a transitive
include on 1.6.5
You can see the version number for the spring artifacts when building spring. Probably something like 3.0.0.BUILD-20091005102426
Maven surefire settings
UnlimitedThreads knocks out threadCount and perCoreThreadCount.
Look at the various "install" scripts to see the version numbers of the artifacts.
Simple, eh ?
And while we're at it: Half of the work with patching these projects has been about fighting with version control systems and custom made build scripts. So please; switch to GIT and maven: I can build any maven project without reading as much as a readme file. Maven allows me as a casual bystander to *easily* make contributions; I never need to read that 4 page "how to build" manual! Git is the only version control tool that respects my right/duty to patch, change and upgrade other projects. It allows me to do all these things without having to *ask* anyone. Repeat after me: Git and maven encourages free collaboration; a key idea of open source, right ?
- ▼ 2009 (5)