Allowing Tests To Fail

The basic idea of unit tests is that you write them once and run them a lot – you don’t check in your changes unless all the unit tests pass.  This is a great idea.

There is one problem with it – it prevents unit tests from being used as a specification of work to be done later.  I don’t mean it prevents test driven development where the tests are written then the code is to make them pass is added and then the tests and code are checked in.  I mean where a particularly complex piece of functionality is going to be added and the best way to ensure that all the use cases are covered is to write unit tests for them all and let them act as the specification (note: this doesn’t work well if you have a non-technical client, but if you are working on off the shelf software it can be very useful).  You need to plan the set of features to include in a release ahead of time so you wind up with a whole heap of new unit tests which are acting as this specification that at some point in the future you will make pass.  At the moment though, those tests are okay to fail.

How do we preserve the ability of unit tests to ensure no regressions while still being able to use them as a to do list?

There are a few options.  The most obvious is to split the tests up into stuff that’s done and stuff that’s not done and make them separately runnable.  The standard build process only runs the “stuff that’s done” set of tests and the others can be ignored until those features are to be implemented.

This however has the disadvantage that if some of the to do tests already pass (the functionality is partially present or is fully available from buggy pre-existing code etc) or if over time some of the tests begin passing because of other changes, it would be nice to ensure that progress is not lost inadvertently.

The solution I’m trying out is to create a list of tests that are okay to have fail for whatever reason and a custom ant task that reads that list and the JUnit reports and fails the build if any of the tests that should pass didn’t but lets it pass even if some of the to do tests fail.  The ant task then tasks the next step and automatically identifies any to do tests that now pass and removes them from the list of tests that can fail.

With that setup once a test passes once it is never again allowed to fail.  Every change is then a step forward.  You can even use the number of tests remaining in the to do list as a rough progress meter if the functionality takes a number of steps to implement or has multiple developers working on it.

It should be interesting to see how this pans out anyway…