CI Isn’t a To-do List

When you have automated testing setup with a big display board to provide clear feedback, it’s tempting to try and use it for things it wasn’t intended for. One example of that is as a kind of reminder system – you identify a problem somewhere that can’t be addressed immediately but needs to be handled at some point in the future. It’s tempting to add a test that begins failing after a certain date (or the inverse, whitelisting errors until a certain date). Now the build is green and there’s no risk of you forgetting to address the problem because it will fail again in the future. Perfect right?

The problem is that it sacrifices the ability to easily isolate the cause of a failure. You can no longer revert changes to get back to a working baseline because the current time is a factor in whether your tests pass or not. You can no longer run historical builds through CI which you may need to do as part of supporting clients on older versions.

It also inevitably leads to false failures as the time points are almost always arbitrary and estimated time lines for completing tasks often slip. So the board goes red and the response is to just suppress it again.

Continuous integration isn’t there to remind you to do something – it’s there to tell you if your build is ok to ship to production or not. It doesn’t make sense to say that a build is ok to ship to production now but not ok to ship in a week’s time. If production is going to blow up in the future because of a bug in the code you want to the board to be red now, not when production actually blows up. And if the client is prepared to accept the risk and leave the problem unfixed then it’s not yet a requirement and doesn’t need tests asserting it – just put a card in the backlog for the client to prioritise and play when required.

If you need to remember to do something, either put a task in the backlog or on the current story board to do it or put a reminder in your calendar. Keep CI focussed on telling you about the state of your code right now – if it’s broken it should be red, if it’s not broken it should be green. Not broken until next week should never be an option.