Continuous Integration Requires Integrating Continuously

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible.  Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days?  Nobody –that’s who.  And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak.  But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

Unfortunately, what Sarah describes is not continuous integration. At best, it’s regular integration but really it’s just automated testing. The key principal of continuous integration is that all developers are merging all changes, all the time. That way you see the messy combinatorial effects of concurrent changes and can fix them before you waste a lot of time going down incompatible paths. Anything that reduces the frequency of doing that integration is a step away from continuous integration, regardless of how often automated tests are run.

Let’s take a look at the basic process they’re using:

Next, we go into development and testing cycles on the story branch.  The developer is making local commits on her workstation.  […]  When she reaches a point where she wants to run tests against her work, the developer pushes those local commits up to the repository –still on the story branch, of course.  Bamboo sees that there’s been a change, and starts building the corresponding Plan branch/es.

This is not continuous integration. The developer’s changes are being kept away from the main team and not integrated while they’re being developed, while they sitting just in the local git repository and while they’re in the story branch. Even though Bamboo runs tests against the code, there has been no integration done here. This is just automated testing, not continuous integration.

It seems that Atlassian have discovered this lack of integration and have developed ways of combating it:

To help find conflict before merging to master, the developer can choose to have Bamboo merge master into her story branch each time she pushes changes to the repo, then run through the CI Plan against the merged code. If the Plan passes, Bamboo will commit the merged code to her branch.

Finally we get some integration happening, but only one way. Since everyone’s off working on feature branches this is really only integrating with completed stories. All the work going on in parallel on other stories is still not integrated so we have no idea if they will work when combined.

Some teams are taking this workflow a step further and requiring developers to merge to an integration branch during development, before finally merging the completed story to master. This practice ferrets out integration issues even faster. Developers get to mash their in-progress stories together several times a day, rather than waiting until one has merged a completed story to master (which may happen only every 2-3 days) and the other has pulled code from master onto their story branch.

The key lesson here is that if we integrate more often, it’s easier to ferret out integration issues. It’s like continuous integration but with the extra fun of managing branches…

Of course, Atlassian aren’t alone in the use of feature branches – it’s become all the rage since DVCS supposedly made branching cheap and merging painless. Unfortunately while DVCS has eliminated many of the basic mechanics of merging, they haven’t done anything to reduce the risk of different developers push the codebase in incompatible directions at a structural level. The only way to avoid that is to integrate all changes as often as possible – that’s continuous integration.

The only way that feature branching and continuous integration can be at all compatible is if the build system automatically builds and tests every possible combination of branches, all the time – a massive waste of build system resources. Anything less leaves potential combinations untested until they are finally merged on to master and all the problems are discovered.

Or you know, we could just do continuous integration…