Coping with Bugs

A little while back Charles Nutter posted about his progress at getting the JRuby bug list under control:

Roughly around September, we crossed an important threshold: 500 open bugs. They dated as far back as 2006 (we moved to Codehaus in 2006), and across the entire release sequence of JRuby; there were even some bug reports against pre-1.0 versions. From September until about mid-January, we worked to keep the number of open bugs below 500. Sometimes it would peek above, but we were generally successful.

About mid-January, I figured it was time to really bite the bullet, and I started walking from the oldest bug forward. It has been a long slog, but I’m making great progress.

Finding the right balance between new features and bug fixes is always really hard. The more experience I get the more I think it’s important to keep the known bug count low. There’s a surprisingly large cost to having bugs sit around unfixed in a bug reporter tool. Even if no one else finds the bug and calls support, each time you try and sort through the bug list to find something, to prioritize bugs or work on stuff you also have to trawl past that bug. As the number of bugs increases it gets more and more likely that they’ll get in the way and the more time you waste because of it.

It gets worse though. Having a lot of known bugs leads to a culture of not fixing bugs because it’s just one more bug so add it to the list.  Getting the count down is always too big a job so it keeps getting put off and the probably just keeps getting worse. If your competitive advantage is based on making users more productive or being easy to use, that’s a pretty disastrous situation because even small bugs dramatically affect the usability and productivity of software.

The right balance between features and bugs is going to be different for each time and each product. I think the key to finding the right balance lies in keeping the number of known issues relatively constant. If the bug count is increasing on average you’re eventually going to get too many bugs to handle and have to go on a fixing spree. A decreasing bug count certainly isn’t a bad sign, but if features aren’t being added fast enough to meet client demands you do still have a problem.

There’s not really a lot of flexibility in trying to keep the bug count stable though. If you’re under pressure to add features faster – how do you manage that without ignoring bugs? Short term there isn’t much you can do. Longer term the key is to reduce the number of bugs you’re adding. After all, if you add fewer bugs you don’t have to spend as much time fixing them and you can focus on features.

What I’m finding useful is the charts plugin for JIRA. I’ve added a couple of simple graphs to my dashboard. One shows the total number of open bugs over time and the other shows the number of bugs opens vs the number closed each month. Together they give a pretty clear indication of the quality trends for the product. It’s not perfect because the levels of testing and deployment isn’t constant throughout the life of a product so the influx of bugs and the amount of time spent on fixing obviously varies but looking at the month long view gives a surprisingly consistent view.