Make your tests fast again !

Spring framework is quit slow at startup stuff, regardless of what top responders on stack overflow seem to think.


Over the years we have developed quite a few tactics to work with this, like starting the application in Unit @BeforeClass or even a JUnit run listener. This sort-of worked with old-style monolithic applications but it was usually a quite poor approach; there are all sorts of reasons for wanting to start slightly different permutations of your application, and why can't just spring be fast ?

As of spring 4.2 you can do this:

  return new AnnotationConfigApplicationContext() {

   @Override protected void resetCommonCaches() {
   }
  }

This will dramatically speed up creation of your subsequent spring contexts.

There are a couple of possible gotchas here; since this will re-use cached type information (cached in static scope) form your previous test run, much increasing the startup speed of a subsequent spring context within the same forked process. Moving all your IT's to a single project is always smart, since class loading and static initialization is the real killer in terms of startup time.

The method in AbstractApplicationContext (the base class) does this:

 protected void resetCommonCaches() {
  ReflectionUtils.clearCache();
  ResolvableType.clearCache();
  CachedIntrospectionResults.clearClassLoader(getClassLoader());
 }

Happy hacking !

Break the bloated library cycle with Java 8

Most of us have, at some point in time, wondered why all successful java libraries tend to "bloat" by adding new stuff all the time. Spring is a great example. Why on earth is there spring-jdbc, spring-transaction or even spring-web ?

Apart from the obvious market-wise need for a companies to appear as a continuous sources of innovation (investors like that), there are some technical issues involved too. This post looks at how java8 can change some of the dependency issues; it may even change how you do java7 right now.

Java7

The dilemma: A class/library wishes to reuse external services, which introduces dependencies.

public class MyClass {
    private ExternalDependency1 ed1;
    private ExternalDependency1 ed2;
   public int doSomething(String input){
       return ed1.calculate( ed2.transform( input));
   }
}

Now this is the classical reason why need to make *our* module depend on something external, ExternalDependency1 and ExternalDependency2 are defined in another project. This is basically the technical reason why we have 280 different "spring" projects; they all tie in with their different dependencies.

Java8 to the rescue

In java8, we can easily move the declaration of the required type *into* MyClass, which makes it self-contained with respect to dependencies:

public class MyClass {

public interface InternalDependency1 {
int calculate(String source);
}
public interface InternalDependency2 {
String transform(String source);
}
private InternalDependency1 id1;
private InternalDependency2 id2;
public MyClass(InternalDependency1 id1, InternalDependency2 id2) {
this.id1 = id1;
this.id2 = id2;
}
public void doSomething(String input){
id1.calculate( id2.transform( input));
}
}

At first glance, this seems like just additionally bloating the code. We need to look at the use before it all becomes clear:

ExternalDependency1 ed1 = new SomeObject1();
ExternalDependency2 ed2 = new SomeObject1();
MyClass mc = new MyClass( ed1::calculate,         ed2::transform);

Now this is where the subtle magic happens; any method that matches the *shape* of the calculate or transform method will be autoboxed into something that can be *used* as InternalDependency1/2.

With this one magic little change, MyClass suddenly lost all its external dependencies.

What about java.util.function.Function ?

Some of you will notice that I am using 1 method interfaces instead of the generic java.util.function.Function class. There are several reasons for this:

A) It is an explicit type that will some day provide IDE support. Current IDE support is somewhat sketchy, but this will improve.

B) "MyClass" is a library component that is expected to have clients; the "cost" of these interfaces making the external contract of "MyClass" 100% clear is probably worth it in the long run.

C) This is java5/6/7 compatible, although the java7 client will have a slightly less cool client api (anonymous inner class galore)

D) If you use Function, the only thing expressing the intention of the function will be the local variable name; in this case id1 and id2. This makes our java code all javascripty; much as I love javascript I am a little sceptical about too this in java; at least for "library" style code.

100 days of Whole30, my personal summary

We interrupt the general nerdiness of this blog with a little experience report of my whole30 lifestyle change. A 30 day "diet" designed to improve your health and maybe loose some weight in the process (http://whole9life.com/).

I started at the end of July, weighing a solid 96,5kg, probably 20cm too short for that weight - and I'm 42 so it's getting sooo hard to get rid of.

Whole30 is about eating natural ingredients, preferably organic. No sugar, alcohol, wheat or anything containing any "E"'s. Just 3 daily meals of real, natural food. I've been 100% faithful in the whole period, eating organic and the highest quality I can find here in Oslo.  All along these 100 days I've been running 7 km three times a week.

Day for day, this is what happened:

Day 5: Out shopping at IKEA with wife, I realize I feel really energized; damn good! Feeling good at IKEA is a first for me, ever.

Day 8: My sense of smell returned 100%.
   Having chronic tonsil & sinus infections, I removed tonsils and surgically opened sinus canals 3 years ago. In those three years, I have had like 25% normal sense of smell (I could wash the toilets with 20% chlorine and not notice it!). All of a sudden I started smelling all kinds of things.

Day 10: My blood pressure goes up.
   I've always been borderline high blood pressure, most of the time *at* the treatment limit. Unfortunately I seem to not tolerate /any/ blood pressure medication, my body seems to reject them each time I use them. So, in line with doctor's orders I start with medicine for enough to get my pressure down (5 days, give or take).

Day 14: I am quite angry
   My wife complains that I am easily agitated. I know this too, but I also feel it is different from the insane rage I could get if I followed something like the cambridge diet.


Day 16: I stop blood pressure medication, back at my "usual" borderline level. (This is according to doctor's orders; please follow your own doctor on this! If I stay on /any/ blood pressure medication for 14 days I collapse)

Day 18: I feel great.

Day 20: My blood pressure drops by 15 (over/under). I have never had this low blood pressure, ever. (This has no connection to the medication I quit a few days earlier, the medication only brings me immediately below the treatment treshold) My new blood pressure is "normal." I've /never/ had "normal"  ever.


Day 25: Wife still complains I'm easily agitated

Day 30: I'm about 5 kg down

Day 40-50: I feel great. My mood is fantastic. Colleagues at work start commenting that I'm in a fantastic mood. They've been informed and curious since day 1, but I think it's becoming clearly obvious to everyone that there is change going on.

Day 60: Wife says I seem to be happy.

Day 70: I feel fucking marvellous, and we're talking mental state. I've never tried cocaine, but this must be what it's like. (Ok, probably not, but this is healthy!)

Day 80: Wife says I seem extremely happy.

Day 100: Officially "done" with the "diet" part. Drinking my first glass of wine & eating sushi. I'm not sure what I'll do next but today I added organic red wine and sushi to my diet; if I can add those two things I'll be satisfied for quite some more time. 10kg down.

There's one thing I'm still unsure about; and that's ketosis. I seem to be going in & out of ketosis all of the time, and I'm probably eating too much or too little carbs. There's a lot of transitional issues involved in going into ketosis (bad breath being perhaps the most notable). It seems like ketosis is good for losing weight, but I assume it won't be necessary now that I'm approaching an acceptable weight.

I know my colleagues will be starting whole30 soon. In thirty days you can find out what impact your current eating has on your health. How can you not take just 30 days ? I took 100 ;)

Your new password

I was fed up with having a zillion passwords. Worst of all, we don't have any insight into what kind of security the websites you access employ, so happy anarchists like LulzSec and Anonymous can run away with your master password if you're not careful.

So instead of a password, I ended up with the following algorithm;

• Second letter in the domain name (Or last letter), in uppercase
• Constant part. Starting with one or more captialized letters, ending with a digit. 6 chars or more. You may choose to look at http://www.baekdal.com/tips/password-security-usability for this bit.
• A digit 1 (increase this for sites that require password change)
• The number of vowels (or consonants, or [A-F] or whatever in the domain name)
• First letter in domain name, lowercase

The constant part should probably contain at least one "special" character like % { or whatever, simply because sites may start requiring this. But think about accessibility for this character on phones etc.

A couple of key points about this algorithm: All digits are grouped together, as for capital letters. Some devices (iPhone etc) require additional touches to switch to upper case or numeric keyboards, so group them together.

The "domain name" is the last word before .com/.org.

I keep the constant part about 6 chars, to make 10 digits total. You might consider going for 12 total.

In some places you may be likely to give away your password (this may happen to me at work sometimes). In those cases you can use a different constant-part or simply a totally different password, at least I always know these things up-front.

Think it sounds like a hassle ? I feel the hassle every time I encounter a site where I haven't used this pattern yet, you get used to it.... This password should be strong enough to last the rest of my life. (Lol, I know that's not going to happen).

Watch that @Autowired

Those who follow me on Twitter, will see that I just submitted a patch for SPR-6870. Great patch if you're running something like a web-based application with lots of non-singleton scoped components. You might save a server or more by applying that patch and the spring context starts quite a bit faster too.

So while you're waiting 20 minutes for spring to compile, just look out for one more thing:

Don't autowire into arrays/collections unless the containing component is a singleton. Spring does not do a very good job of caching that stuff (it sucks, actually).

So if your non-singleton scoped classes (or junit tests - like we had ;) contain something like

@Autowired
private MyFud[] myFuds;

Just make a separate singleton object that encapsulates the array/list and wire that into your controller class / junit test instead. Bang you're safe.

Maven Surefire 2.7.1

Todays release of surefire 2.7.1 is an important milestone for me. Lets start out with what the 2.7 series has to offer

• Multiple run-orders for tests now supported
  The runOrder attribute lets you specify alphabetical, reversealphabetical, random, hourly (alphabetical on even hours, reverse alphabetical on odd hours) and filesystem. Odd/Even for hourly is determined at the time the of scanning the classpath, meaning it could change during a multi-module build.

• Faster, smaller
  About 1/3 of the total download size of 2.6. I get quite a lot of feedback saying it's significantly faster too; your mileage will vary.
  

• Parallel JUnit
  Surefire is now totally self-contained and no longer uses ConfigurableParallelComputer for anything. Real execution times per test are also reported. It won't get much better than this.

• Severe memory/resource leak fixed for those of you who have console output.
  The more your tests were writing to stdout/stderr, the worse the problem was. This one has been here since 2.4 ;)

• Pluggable/Selectable providers
  Surefire is a framework for forking and reporting with a few additional features, such as directory scanning services. Until 2.7, this has all been wired together in one monolithic slab of code with test-providers ( TestNG, junit3/4/4.7) seemingly independent but in reality all welded together by massive dependencies and strange divisions of labor. No more. 2.7.X makes it possible to write your own providers. Best of all there's really not much work you need to do to create one.
  Need a "fork every 20 tests" provider? Fork one of the existing ones and make it yourself on github, probably in less than an hour. This also means we will be closing some of the more exotic requests as won't fix, since you can just do it yourself.
  Read about it here and here for the api

• JDK 1.3 fork-compatibility restored.
  I ended up doing this just for the sheer heck of it; kind of a challenge. Pardon the language.

There's other issues fixed too, but these are the highlights.

* Why ?
I started programming on my C64 when I was 13 and I've been coding passionately ever since. I went to university and when I finished developing software became my day job. And although I keep my code clean, professional software development is also a lot about making deliveries with trade-offs and sometimes compromises. And we move on. If it doesn't come back to haunt us, it was probably good enough - no matter how frustrated you felt when making it.

Not so with my Open Source work. There I will only do stuff that somehow is the most excellent work I am capable of. It's the hobbyist computing returned, but my capacity is oh so infinitely different from when I was a teenager. Several OSS companies have offered me jobs, which I have turned down. If I was to take that I somehow feel I'd be taking the hobby as work once more.

So how does all of this relate to surefire? Until recently, surefire has been real messy code. While the basic design contains some interesting and pretty sophisticated stuff, it was basically a big mess. Huge classes with 400 line long methods is the trending topic. It was code without any discernible shape or form; mutable state mutating at every opportunity. All this mostly due to different people working on it at different points in time.

I've seen this happen, even with the best of people working together, and I will not dwell further as to the reasons for this. But I love working with the structure of code, and I think programming is communication; me talking to you - disconnected in time and space.

So I refactor. And I think. Sometimes I can think about a change for a week, code it in few hours only to realize it was wrong and throw it all away. Sometimes I can discover midway that there is a more important angle on solving the problem; something totally different is more important and will have bigger impact on future change. And I just stash away everything and do The Thing I Now Know Is Right. Repeat until satisifed.

I spent months of spare time working with surefire 2.7 this way. I think the results are pretty amazing; the plugin is basically transformed into clean code. Every year around Christmas time, I tend to bring this book to the fireplace. It's still one of the most amazing books written about how to think code. Kent has rewritten this book for Java several times, but I still recommend the Smalltalk version - even if you never programmed smalltalk. And take a look at surefire trunk. There's still work to do. Make a patch. Merry xmas.

Concurrency in maven ?

Within the maven community, there has been a push towards parallelizing maven itself
to achieve better build performance within multi-module reactor builds. A number of strategies have been tried, and the main two strategies are: Parallel reactor mode (uses module dependency graph to schedule builds that can be built in concurrently side-by side). The other strategy is known as "weave" mode, and it traverses the modules phase-by-phase instead of module-by module (you can read about it here)

Both have "fully" functional implementations available, and weave mode is quite a lot faster than parallel mode. The code is available at http://github.com/krosenvold/maven3. Just build and run with -Dmaven.threads.experimental=4

So what is this post about ? I am the primary author of "weave" mode, and for the last weeks I've been searching for an elusive goal: 1000 consecutive green builds of 1 single project on my CI environment.

Initially I was quite afraid of the thread safety issues withing maven; after all retromounting concurrency to any non-concurrent code can be a daunting task. Fortunately there is a lot of state that is /copied/ in maven reactor mode. From a concurrency perspective, this saves the day.

So why am I not getting my 1000 greens ? Every 3-400 builds it would fail, with strange errors. I asked a few questions (and this one) on stackoverflow.com. It's the file system. The java file system
has no guarantees of /anything/ when it comes to concurrency. The only thing you can be sure about is that the single thread that wrote the file can also read it afterwards.

javac uses the file system. And I was quite baffled by this; in weave mode the javacs are invoked on a pretty tight schedule; they typically come within just a few ms of each other. Every now and then the downstream javac would complain about "bad class files" from the upstream javac. But the scheduling is done properly, and the first javac /was/ done. How to solve it ? Turn on "forkMode" in maven for javac.

I had a chat with the nice folks at #kernel and they told me that all contents of a file should be concurrently visible to /everyone/ upon close() in a modern linux kernel. When I turned on forkMode in javac the problem went away. Because forkMode=true basically delegates the visibility issues to the os.

You /can/ try this yourself if you check out from github and try to build a project. It works best if you do a "mvn -Dmaven.threads.experimental=4 clean install", since that'll write a lot of files.

I'm still scratching my head about what to do with this; given that forking delegates visibility to the underlying os one could just fork everything all the time. Or find some other option.. Suggestions ?