Are four cores required to surpass one ?

My recent post about running unit tests in parallel has led to some interesting food for thought:

When running regular unit tests, running with 2 cores seems to be no faster than one core. 4 cores (and arguably a newer CPU architecture) had a significant impact, where performance nearly doubled. The hyperthreaded quad-core i7 920 was *slower* than a 3Ghz core 2 duo when running single threaded, but nearly twice as fast in a multithreaded scenario. What's happening ?

I looked into the issue with jprofiler, but there is no obvious lock contention or other clearly visible bottleneck. The only explanation I can think of so far is this:

The test I was running was extremely simplistic, and there was no real cpu requirement in running the test methods. So the overhead of creating thread pools and all that actually amounts to just as much computation as running the tests. Likely ?

After my summer vacation, I will try to find out. I probably need to run with more realistic test cases. I figured I'd build surefire with threads, because thats a fairly heavy build. In the mean time, your thoughts are welcome.

From legoland,

Kristian

Run your junit tests in parallel (with spring 3.0 and maven)

Swimming in a sea of patches...

Note; Most of the content in this post has been surpassed by a newer entry.

If you're willing to apply a few patches, you can now run your junit tests in parallel. There are two main use-cases for this; one is to make your regular unit tests run faster, the other is for io-bound tests, where you want to run several in parallel because you're waiting for external systems.

Just to whet your appetite, here are some numbers from a testcase (running 1000 instances of a simple unit-test-class):

Legend:
SC = SingleThreadedComputer (Computer, Junit47-snapshot class)
PC = ParallelComputer (ParallelComputer, Junit4.7-snapshot class)
CPC = ConfigurableParallelComputer (My project, see references)
C=Classes (one thread per class)
M=Methods (one thread per method)

	Core 2 duo 3.14Ghz	Intel i7 920
SC	203	451
PC(Classes, Method, Unlimited threads)	406	806
CPC(Methods,16 threads)	218	214
CPC(Classes, Unlimited threads)	203	116
CPC(Classes,Methods, Unlimited threads)	266	147

Running time in ms, 1000 simple tests

The table shows that for most dual-core solutions, running with parallel threads is currently never any faster than running non-threaded. Switching to the brand-new Intel i7 (4 hyperthreaded cores) has a marked effect, where simple unit test performance is doubled.

The lack of punch does, of course, change the moment your tests start doing IO (Integration tests, selenium tests and other web-tests).



The juicy stuff
svn co https://svn.openqa.org/svn/selenium-rc

Second build with mvn clean install: 102 seconds.
With 80 threads class-parallel on the i7: 37 seconds.

Class-parallel works easily on most tests.

A sea of patches

Getting all of this to work was not without its problems, though. To get this running, I had to patch just about half the libraries in use in my current stack. Furthermore, there's a huge difference between getting it to run and getting it to run without hiccups - every time. These patches insure that your infrastructure works as it should, but are your tests concurrent ?

Patches involved

See below for how to apply these patches in a simple manner.
Spring (3.0 trunk):
http://jira.springframework.org/browse/SPR-5863
AspectJ: Patch trunk with this patch (if using spring) (This patch is included in version 1.6.6)
https://bugs.eclipse.org/bugs/show_bug.cgi?id=281654
Junit4.7:
Several patches needed for thread safety:
http://github.com/KentBeck/junit/issues#issue/16
http://github.com/KentBeck/junit/issues#issue/18
Maven: You need a patched version of surefire that supports junit 4.7:
http://jira.codehaus.org/browse/SUREFIRE-555
You'll also probably want to be using the ConfigurableParallelComputer instead of the ParallelComputer that's included with JUnit.
git://github.com/krosenvold/configurable-parallel-computer.git

With these patches in place you're smooth sailing ;)

Applying the patches

Note: If you did this on the pre rc1 version of this patchset, I recommend that you start from a fresh copy of everything this time.
If you're using maven you could try this sequence:

git clone git://github.com/krosenvold/spring-framework-concurrent-junit-patch.git
git clone git://github.com/krosenvold/junit.git
git clone git://github.com/krosenvold/configurable-parallel-computer.git
git clone git://github.com/krosenvold/spring-test.git
git clone git://github.com/krosenvold/spring-concurrency-test.git
cd junit
ant
cd ../spring-framework-concurrent-junit-patch
./installJunit
cd ../spring-test
mvn install
cd ../configurable-parallel-computer
mvn install
cd ../spring-framework-concurrent-junit-patch
./checkoutSurefire
./buildSureFire
./applySureFirePatch
./buildSureFire




cd ../spring-concurrency-test
mvn install # Test project to verify that everything is ok


Your new versions

JUnit should be modified to 4.7-CONCURRENT
spring-test should have version number 3.0.0.RC1-CONCURRENT
Surefire should be modified to 2.5-SNAPSHOT
You must update your aspectj version to 1.6.6 in your project. Spring does a transitive
include on 1.6.5
You can see the version number for the spring artifacts when building spring. Probably something like 3.0.0.BUILD-20091005102426

Maven surefire settings

          <plugin>
              <artifactid>maven-surefire-plugin</artifactid>
              <version>2.5-SNAPSHOT</version>
              <configuration>
                  <parallel>classes</parallel>
                  <useUnlimitedThreads>true</useUnlimitedThreads>
                  <threadCount>80</threadCount>
                  <perCoreThreadCount>true</perCoreThreadCount>
              </configuration>
          </plugin>

UnlimitedThreads knocks out threadCount and perCoreThreadCount.

Look at the various "install" scripts to see the version numbers of the artifacts.
Simple, eh ?

And while we're at it: Half of the work with patching these projects has been about fighting with version control systems and custom made build scripts. So please; switch to GIT and maven: I can build any maven project without reading as much as a readme file. Maven allows me as a casual bystander to *easily* make contributions; I never need to read that 4 page "how to build" manual! Git is the only version control tool that respects my right/duty to patch, change and upgrade other projects. It allows me to do all these things without having to *ask* anyone. Repeat after me: Git and maven encourages free collaboration; a key idea of open source, right ?

http://twitter.com/krosenvold