The Curse Of Testing Text

One of the major challenges in my job is testing our product. Now most people think that testing is reasonably easy but requires discipline, this is not true if the product you write happens to be a styled text editor and it's nearly impossible to do really well if you're working with something as flexibly defined as HTML.

The problems start at the unit testing level. Try taking the standard JTextPane class and writing unit tests for it. How do you test that it can render a HTML list correctly? You could write a test to make sure that the list numbering at least comes out in order and in the right format but that still won't guarantee that the list numbers actually paint correctly. For instance, we recently had a bug where the list numbers painted correctly if the list fit on screen, but if it required scrolling, whatever item was at the top of the screen was numbered 1 even if it was actually the third item in the list. Our list numbering unit tests had no way of picking up on that because it was the actual rendering code (which we didn't write) that was wrong.

So you may be thinking that integration tests should have picked up that bug. Not a chance. How do you write a set of tests for the list rendering that doesn't break because we added a slight padding to the body which caused the list to indent a little more than it used to (but still render correctly).

The number of false negatives you get from automated tests is staggering and makes the tests completely worthless because adding new features or fixing bugs (like getting padding to work correctly) causes them to fail.

So what about manual testing? It's not particularly effective either. Certainly it would (and did) pick up the list numbering bug I mentioned, however it didn't pick up the other list bug we've had reported recently. To reproduce:

1. Start with a document that contains just an ordered list with 4 items.
2. Select and indent items 2-4.
3. Select item 4 and outdent it again.
4. Select item 3 and outdent it again.
5. Delete item 3.

Now tell me, when was the last time you saw a testing plan that would have included something like that? That's certainly not the strangest bug I've seen either. The trouble is that every action that's performed in the editor modifies the document state and that modification then affects every other operation that occurs in the document. There's simply no way to isolate the operations on the document completely short of wiping the users document to get a clean document all the time (which doesn't exactly make for a useful editor).

Lets take a look at something relatively simple – inserting a HR. Here's what I'd go through to perform basic testing:

1. Insert a HR into a blank document.
2. Insert a HR at the end of the first and only paragraph in the document.
3. Insert a HR at the end of the first of two paragraphs in the document.
4. Insert a HR in the middle of a paragraph.
5. Insert a HR at the end of the first and only item of a list.
6. Insert a HR at the end of the first of two items of a list.
7. Insert a HR at the start of the first and only item of a list.
8. Insert a HR in the middle of a list item.
9. Insert a HR into an empty list item.
10. Insert a HR into an empty table cell.
11. Insert a HR at the end of a paragraph in a table cell.
12. Insert a HR at the start of a paragraph in a table cell.
13. Insert a HR in the middle of a paragraph in a table cell.
14. Insert a HR into an empty list item in a table cell.
15. Insert a HR at the start of a paragraph in a list in a table cell.
16. Insert a HR at the end of a paragraph in a list in a table cell.
17. Insert a HR in the middle of a paragraph in a list in a table cell.
18. Insert a HR into an empty table cell inside an empty table cell.
19. Insert a HR at the start of a paragraph in a table cell in a table cell.
20. Insert a HR at the end of a paragraph in a table cell in a table cell.
21. Insert a HR in the middle of a paragraph in a table cell in a table cell.
22. Insert a HR without first clicking in the editor to position the caret.
23. Insert two HRs in a row.

23 tests for inserting a HR, seems reasonably comprehensive right? Nope, if you inserted three HRs in a row it deleted the document (undo would recover it though). We also haven't taken into account inserting into arabic text and making sure the text direction remains correct. Now think of how many tests you'd need to cover inserting a table and all the different things you can configure about a table (number of rows and columns, height and width as pixels or percentages, background colors, border size, alignment etc etc). Then think about lists, images, bold, italic, underline, strike through, sup, sub, merging cells, splitting cells, deleting things, selections spanning multiple blocks and a ton of other features. Then think about trying to parse and correct random HTML that's thrown at you.

Considering all of that, how do you manage to quickly fix issues reported by customers and send them a new build that's been well tested? How about when you have hundreds of new customers going into production each month? Did I mention you have an engineering team of 4?

The fact is, this stuff is incredibly hard. There's a good reason that I laugh at people who think they can just whip up a HTML editor cheaper than our licensing fees, we've worked through all these issues – found them, fixed them and learnt how to engineer our code so that we're confident they work without testing them. Testing is critical obviously, but reducing the number of errors in the code originally is even more important.

Trying to salvage some real point to this post: testing tools for text are practically non-existent and I'm dying for a way to actually automate these tests. If you're up for a challenge, maybe testing text would be a good one to try.