What Is Included In The 10 Minute Build?

Having a fast build is an important part of XP so that the continuous integration cycle doesn’t take too long. Do people usually see this as including all the acceptance tests as well or does this just include the developer’s tests?

How do people handle acceptance tests failing? Does it cause the build to fail or not? What about acceptance tests that have been added for the current iteration but haven’t been implemented yet?

At Ephox, we treat acceptance tests as critical tests that must never fail and we run them on every build as part of our continuous integration. We do however have an in progress directory where we put acceptance tests that are yet to be implemented and currently our build system doesn’t run them at all. At some point we’ll update the build script to run these tests, probably as part of the nightly build, and report on which are passing and failing. We also need to get more into the habit of monitoring these tests and making sure they are moved into the acceptance tests proper once they are passing.

I need to think a lot more about how to write acceptance tests as well – our current tests are written from the user’s point of view, so they are end to end tests and tend to be slow to run. The advantage is they comprehensively test what the user expects, instead of just looking at part of the systems internals and assuming that the full code path works. There seems to be advantages and disadvantages with both forms, but I do find that having end to end acceptance tests makes me feel safer. Having efficient atomic and integration tests that test the expected internals for those acceptance tests lets me work fast with a high level of confidence because in most cases if the atomic tests pass, the acceptance tests will too. It’s nice to have the acceptance tests making sure nothing falls through the cracks of the atomic tests though.