Manage to Test

Michael Bolton, along with James Bach created the Rapid Software Testing class from which this session was greatly condensed. I’ve never taken the course, but I’ve read a lot of Michael’s writings and one of my my past employee’s – Eric Jacobson – took the class, so I knew most of the material going in. I’m glad that there were few surprises – showed that maybe I understood what I’ve read! That being said, there were some choice points that I was able to synthesize with my experiences over the past eight years.

You don’t ever have “not enough time to test.”

Testers often complain that they never have enough time to get all the testing done. We should know that it is impossible to get all of the testing done. For example, try using as inputs all the possible values between 0 and 1. Now 1 and 2. And so on. But we certainly have enough time – we have all the time we’re given. And that’s enough to test as much as we can within that time, by definition.

It sounds a bit flip, and I guess it is. I tried that with my management a few years ago when they asked me how long it would take to test something. I responded that we’d test as much as we could given the time allotted. They were never really pleased about me answering in that way. I don’t blame them – it’s frustrating when other people don’t have an answer that you can fool yourself into thinking is real and then turn around and blame those people when that answer turns out wrong. (Read that again, I’m pretty sure it makes sense!)

Nowadays I don’t predict how long it will take to test a release or a feature, I predict how much work a team will complete (and that includes testing) in any given period of time based on past performance, in 2-4 week increments. But that’s another story.

“Bugs prevent you from finding bugs.”

Any tester should know this – the act of examining, reproducing, and writing bugs prevents you from more testing. If there are a lot of bugs, you can’t complete much testing at all! The takeaway from this class is that makes it very important that you test as rapidly as you can to counteract this effect. That’s true. And the approaches and techniques discussed in the class were invaluable – I recommend the class!

But testing faster isn’t the total solution. We also need to stop producing bugs in the first place. The best way I’ve found to do this is Acceptance Test Driven Development (or BDD, or Executable Specifications). Write your story/spec and include examples – examples that can be run to verify functionality and to use as regression. I should be quick to mention these aren’t acceptance tests, but more appropriately “rejection checks.” With these examples, we’re not defining the spec, but rather framing it. They don’t stand alone and are not independent of the story they describe. Take the “Petals on the Rose” dice game. Teams, not knowing the rules, come up with a number of passing tests, and often it leads to an isomorphic version of the specification, if not the actual ruleset. These tests would be a fine set of rejection checks for the

story but the story – the ruleset for the game – still has to be there.

“Testing is not about probing for functional correctness.”

That’s what the above Acceptance Tests are for; they make sure the system does what we, and the customer, expects. And unit testing is for making sure the code works. Testing is looking for problems we don’t expect. We don’t need to learn about expected functionality – we already know about that. Testing is about finding the problems that may lead to loss of customer value. Things we *don’t* expect. If we can anticipate these things and get them included as part of our stories and acceptance tests, fine. But sometimes we need to actually see/touch parts of the working system to discover the presence and possibility of such problems. That’s testing.

And that’s what I synthesized today.