Michael Bolton, along with James Bach created the Rapid Software Testing class from which this session was greatly condensed. I’ve never taken the course, but I’ve read a lot of Michael’s writings and one of my my past employee’s – Eric Jacobson – took the class, so I knew most of the material going in. I’m glad that there were few surprises – showed that maybe I understood what I’ve read! That being said, there were some choice points that I was able to synthesize with my experiences over the past eight years.

You don’t ever have “not enough time to test.”

Testers often complain that they never have enough time to get all the testing done. We should know that it is impossible to get all of the testing done. For example, try using as inputs all the possible values between 0 and 1. Now 1 and 2. And so on. But we certainly have enough time – we have all the time we’re given. And that’s enough to test as much as we can within that time, by definition.

It sounds a bit flip, and I guess it is. I tried that with my management a few years ago when they asked me how long it would take to test something. I responded that we’d test as much as we could given the time allotted. They were never really pleased about me answering in that way. I don’t blame them – it’s frustrating when other people don’t have an answer that you can fool yourself into thinking is real and then turn around and blame those people when that answer turns out wrong. (Read that again, I’m pretty sure it makes sense!)

Nowadays I don’t predict how long it will take to test a release or a feature, I predict how much work a team will complete (and that includes testing) in any given period of time based on past performance, in 2-4 week increments. But that’s another story.

“Bugs prevent you from finding bugs.”

Any tester should know this – the act of examining, reproducing, and writing bugs prevents you from more testing. If there are a lot of bugs, you can’t complete much testing at all! The takeaway from this class is that makes it very important that you test as rapidly as you can to counteract this effect. That’s true. And the approaches and techniques discussed in the class were invaluable – I recommend the class!

But testing faster isn’t the total solution. We also need to stop producing bugs in the first place. The best way I’ve found to do this is Acceptance Test Driven Development (or BDD, or Executable Specifications). Write your story/spec and include examples – examples that can be run to verify functionality and to use as regression. I should be quick to mention these aren’t acceptance tests, but more appropriately “rejection checks.” With these examples, we’re not defining the spec, but rather framing it. They don’t stand alone and are not independent of the story they describe. Take the “Petals on the Rose” dice game. Teams, not knowing the rules, come up with a number of passing tests, and often it leads to an isomorphic version of the specification, if not the actual ruleset. These tests would be a fine set of rejection checks for the

story but the story – the ruleset for the game – still has to be there.

“Testing is not about probing for functional correctness.”

That’s what the above Acceptance Tests are for; they make sure the system does what we, and the customer, expects. And unit testing is for making sure the code works. Testing is looking for problems we don’t expect. We don’t need to learn about expected functionality – we already know about that. Testing is about finding the problems that may lead to loss of customer value. Things we *don’t* expect. If we can anticipate these things and get them included as part of our stories and acceptance tests, fine. But sometimes we need to actually see/touch parts of the working system to discover the presence and possibility of such problems. That’s testing.

And that’s what I synthesized today.

What are your views for QA integration in an Agile environment?

That was a question posed to me via email. This was my response:

Testers are part of the team. They should not be thought of as a different group and somehow apart. Testing is their primary role, but they should contribute in any way they can to help the team. That means analysis, configuration, coding, whatever. Many struggle a lot with the concept of blurring the lines between testers and programmers. I think it would be great if testers could be writing code, specifically pairing with developers, writing test code while programmers write feature code. Most places I know of don’t have testers with that expertise. Testers don’t need to be able to code to contribute as an integral part of any team.

Here’s my current view on how to do it:

• Testers should be greatly involved in the story creation process with the customers/product owners. Not only is it important that they understand what the customers want, but at this stage they should be able to point out testability of stories and help contribute to the Acceptance tests associated with the story. It is important that the tester *drives* this Acceptance test creation. It’s important for all to understand that just because they are tests doesn’t mean the tester is responsible for them! They aren’t – that’s up to the customer, or whoever is accepting the output of the team – but the tester can make sure that they are written, that they are relatively comprehensive, and that they are testable. For more information see Elizabeth Hendrickson‘s article on ATDD and Gojko Adzic‘s books on Executable Specifications
• Testers should enage frequently with programmers with what I call Dev Shadowing. Not quite pairing, the tester should sit with the programmer as they write code. That brings the testers’ skill to bear as they help the programmers with logic, tests, or anything they create. I know that many testers are apprehensive for fear of looking incompetent or resentment from programmers. I’ve found through experience that these fears are largely unfounded. Over time, as trust is built, perhaps this effort can be scaled back…as the lines blur.
• Testers, at the sprint planning stage, should determine which of the Acceptance tests, and surrounding tests, if necessary, they will implement and which the programmers will implement. If you’re using some kind of Fit-style framework, this could be a collaborative effort where the tester writes the tests and the developer writes the necessary fixtures. Or, if skilled enough, the tester could do it all. If you’re not doing something like this, the tests should be written using a unit test framework.
• Testers, again at the sprint planning stage, should be clear on which tests need to be at the UI level and which are purely related to business logic (which should exist at a mi imum in the UI). If the test doesn’t need to happen at the UI level, it is likely best handled as an automated Acceptance test (by that, I mean a test written using the unit test framework, or using a Fit-style framework). If it must be tested at the UI level, then the tester must decide if it needs to be automated or should be done manually. If you want to automate it, then hopefully you have the tools in place and the expertise and the appropriate story points/work effort assigned to account for the automation. If manual, it’s likely that BAs, and other members of the Product Owner team could assist with this effort. After all, it may just be manually running the Acceptance test as written which is up to the Customer to approve anyhow. This type of testing amounts to validation/verification and can be characterized as checks.
• Testers should engage in exploratory testing. This type of testing goes beyond the Acceptance tests, but not necessarily to their exclusion. This is the area where a trained tester can provide great value over and above the manual testing you get from other members of the team. Session-based testing I’ve found is a good method of organizing this effort so that everyone knows what areas of the system were covered in a given session.
• All tests should be run on checkin! If this is too time-consuming, at least all unit tests should be run at checkin (they should be very fast!) and all unit, automated functional and acceptance, and all automated UI tests should be run on migration to the various server environments, and perhaps nightly or on-demand. Obviously, the trick here is to make sure the tests are robust enough to not overload you with false negatives — but that’s better, imho, than the alternative.

So that’s what I would suggest in terms of integrating testers into an Agile environment. I know Lisa Crispin and Janet Gregory, respected members of the community, have published a book on this very topic.

That’s my new title.

For the last 10 years or so I’ve been a “Quality Assurance Analyst” and a “Quality Assurance Manager.” We all know now, or should, that those roles have nothing to do with quality assurance. I didn’t write feature code as a QA analyst and I didn’t dictate to programmers how to do their jobs (well, of course I did both of those things — with varying levels of success/influence — but it wasn’t part of my job description). As a QA manager I was a “resource” manager; I wasn’t responsible for teams or projects, but rather a group of testers that worked on various projects. The primary responsibility in that job description was to be aware of the availability of the people on my team through forecasting and monitoring so that when new work came up, I could assign someone to do it (and I did everything I could to make this responsibility moot).

So I was a Tester and then a Test(er) Manager. And while I did other things, my focus was always on testing – how to do it better, how to do it earlier, and how to get the most appropriate information possible to the decision makers on projects. I would not, however, characterize my current title as “Test Architect.” I’m not solely focused on testing. I’m also focused on business analysis, and programming, and sales, and support, and management and the SDLC and…well everything we do that has an affect on the quality of our work. While I attempted to have this kind of influence in my former role, I couldn’t. I didn’t have the authority, and often, the support, of others in the organization. Now, I do.

Here are some of the things I’m doing now:

• Leading an Agile transformation
• Helping the business understand internal capabilities and identify needs in the marketplace
• Helping teams coordinate efforts between local and off-shore employees
• Helping dependent project teams work together without hindering and/or blaming each other (it’s nice to finally be able to make a difference in something like this…)
• Modifying the organization structure to better facilitate small, permanent teams of cross-functional developers (analysts, testers, programmers, et. al.)
• Implementing an Executable Specification framework (with the full knowledge that they ultimately function as rejection checks)
• Integrating and aligning Security and Performance teams with project teams

I still have issues with the word “assurance” but I know that I have a lot more influence on Quality – using a variety of definitions – than I’ve ever had before.