Eric Jacobson: I have a question for your blog.
EJ: I just got my ass kicked trying to convince a project team to stop calling features done without the automation done.
ManageToTest: That’s illegal! Call HR!
EJ: I had like 10 people telling me I was wrong.
MtT: That’s interesting. So why is it wrong? Does it not need to be done? What will you do in the meantime? When will there be time to do it if not now? Why is time then “less valuable” than it is now?
EJ: Their bottom-line excuse is that automation is important, but not part of the “critical path.”
MtT: Oh really? Critical path for what? This particular feature?
EJ: They say, “Yeah, it would be nice to have it, but we can go to prod without it.”
MtT: Ah, yes, of course.
MtT: Sure you can put it into prod, but what about the next feature? When will you do automation for that? Oh, well, I guess you can put that off too.
MtT: So when will you do the automation from that first feature? Is there ever a time there aren’t any features?
MtT: What is the value of having this automation? If there is none, then they are correct. Don’t do it.
MtT: But perhaps we aren’t understanding what the value is.
MtT: The value should be the safety net of knowing that new functionality hasn’t broken existing functionality. Right now the only way to do that is by doing manual regression, and the time it takes to do that will increase over time.
MtT: So if you don’t automate, you’ll have to cut some of that manual regression out, creating risk.
MtT: That amount of risk is what you need to determine. That is the “value” of the automation.
EJ: …great stuff. I wish I would have said it in my meeting.
MtT: So you’re saying I should blog about this?
Hey Mr. ManageToTest, do you happen to have any standards or guides around Quality Control Test Plans? We have a few different QC groups doing things slightly different ways….no formal documentation on which way is the “best” way, or best practices around how to write a test plan.
First off, there’s no “best” way to write a test plan. Depending on the context, there will be more effective methods. Also, you probably need to define what you mean by a test plan, and what it is you want to accomplish with it as the definitions and needs vary between contexts.
That being said, I can provide some examples of test planning based on my experience:
I find this method to be wasteful and often redundant. Such plans are written and then never referred to again, and as projects change, these documents are rarely updated so they present an inaccurate record of what was actually done.
A search on your company’s document repository will probably yield several examples of such a document or template, so if you really want to use this style you probably already have it in practice somewhere in your organization.
In many cases, a checklist of capabilities and areas to test are appropriate. This is an excellent way to lay the groundwork for a testing session or the creation of a set of test cases. It is complementary to discussions of user stories when eliciting acceptance tests and during delivery story discussion when eliciting tasks and estimation of work. Here’s a great article from James Whitaker (Director of Testing at Google) where he describes a method of creating and organizing such a plan.
Another method is Session Based Test Management (SBTM) created by James Bach. A Charter is written that describes the goal of the testing session. A list of Charters defines the overall plan for the testing effort (and some predictive value based on the number of charters). Note that this method is a hybrid of a test plan and test cases. The act of engaging in the testing session produces the test cases on-the-fly resulting in a record of what was actually tested. The Charters completed represents your test coverage.
Any one of these ways may be effective in a particular context, or perhaps a combination of things from each of them would suit better. And there are certainly other ways of generating and presenting test plans. Again, I would stress that no method is the “best” across all contexts.
Me – “The team is responsible for Quality!”
Dev Mgr – “Yeah, I hear that, but what will help is to go one level deeper – let’s realistically identify what can and should be done by “developer” and “tester” resources.”
Lots of managers and their employees (and HR) struggle with this concept of team responsibility.
Clearly there will sometimes be tasks on stories that involve a level of coding skill that can only be done by someone with the title of Developer. There also will be some tasks that would be much more efficiently done by someone with years of testing experience. This doesn’t change the fact that when you commit to stories in a sprint, you’re expected to get those stories done. It doesn’t matter who is actually available to do the work.
Let me explain.
In the beginning of each sprint, the delivery team takes “Ready” stories from the backlog and works through them and creates tasks and estimates them. The team should know what their capacity is for that period of time. This means you should have an idea of how much time it will take to code and test and otherwise produce that work. All of it, not just the coding. If people are unavailable that sprint, clearly you should take on less work. If some of the test-related tasks don’t fit within the capacity of the testers you have on your team, you have a choice: either remove some work from the sprint and use the extra time for the programmers to do training and/or help with other projects outside of the team OR have other members of the team take on the remaining testing tasks.
If testing, or any other specialization on the team, is causing a bottleneck, the answer isn’t to shove more work into the bottle! What you do is work together to keep the flow going. If that means programmers doing more testing, fine. If that means testers doing more tasks that traditionally fall to programmers, fine. If that means analysts doing the same, or any other permutation, fine.
If something doesn’t get coded – that’s not a single programmer’s fault. The team should have known about it. If something doesn’t get tested, that’s not a tester’s fault! It’s the team’s fault for not making sure it got done. Blaming isn’t a productive method of getting things done. We succeed as a team and we fail as a team. And when we succeed, we reinforce and build upon the things we’re doing right. When we fail, we recognize the reasons for the failure and work to improve for the next sprint.
Let’s go back to the discussion – assume I said all that stuff above.
Dev Mgr – “Alright – so we come up with tasks together and then assign them to people – that will lock them down and we’ll be able to tell Project Management what resources we have free, or what resources we need to get the work done in this sprint.”
Me – /sigh
Note that making sure you have the capacity isn’t the same thing as assigning tasks to people. Tasks, optimally, should be pulled by people capable of the work only at the point in time that they can start working on them. It isn’t efficient to have a task assigned to someone who’s working on something else while another person could be doing that task. Now you may *know* simply because of the nature of the task that a particular individual has to do it. That’s fine. But be aware that’s risk you’re taking on. If nobody else is capable of that work, then you have to be very sure that particular individual has the capacity in that sprint to do that work. If you have a lot of work like that, it would be advisable to pair that person with someone to work on the tasks together. After a time, then you’ll have two people capable of doing that kind of work and you’ve lessened your risk!
Me – “So you’ve got it now, right? You’ve got a cross-functional team that self-organizes. You don’t need to assign tasks to anyone or wrangle with Project Management over time estimates and the availability of people.”
Dev Mgr – “Resources.”
Me – /sigh
Michael Bolton, along with James Bach created the Rapid Software Testing class from which this session was greatly condensed. I’ve never taken the course, but I’ve read a lot of Michael’s writings and one of my my past employee’s – Eric Jacobson – took the class, so I knew most of the material going in. I’m glad that there were few surprises – showed that maybe I understood what I’ve read! That being said, there were some choice points that I was able to synthesize with my experiences over the past eight years.
You don’t ever have “not enough time to test.”
Testers often complain that they never have enough time to get all the testing done. We should know that it is impossible to get all of the testing done. For example, try using as inputs all the possible values between 0 and 1. Now 1 and 2. And so on. But we certainly have enough time – we have all the time we’re given. And that’s enough to test as much as we can within that time, by definition.
It sounds a bit flip, and I guess it is. I tried that with my management a few years ago when they asked me how long it would take to test something. I responded that we’d test as much as we could given the time allotted. They were never really pleased about me answering in that way. I don’t blame them – it’s frustrating when other people don’t have an answer that you can fool yourself into thinking is real and then turn around and blame those people when that answer turns out wrong. (Read that again, I’m pretty sure it makes sense!)
Nowadays I don’t predict how long it will take to test a release or a feature, I predict how much work a team will complete (and that includes testing) in any given period of time based on past performance, in 2-4 week increments. But that’s another story.
“Bugs prevent you from finding bugs.”
Any tester should know this – the act of examining, reproducing, and writing bugs prevents you from more testing. If there are a lot of bugs, you can’t complete much testing at all! The takeaway from this class is that makes it very important that you test as rapidly as you can to counteract this effect. That’s true. And the approaches and techniques discussed in the class were invaluable – I recommend the class!
But testing faster isn’t the total solution. We also need to stop producing bugs in the first place. The best way I’ve found to do this is Acceptance Test Driven Development (or BDD, or Executable Specifications). Write your story/spec and include examples – examples that can be run to verify functionality and to use as regression. I should be quick to mention these aren’t acceptance tests, but more appropriately “rejection checks.” With these examples, we’re not defining the spec, but rather framing it. They don’t stand alone and are not independent of the story they describe. Take the “Petals on the Rose” dice game. Teams, not knowing the rules, come up with a number of passing tests, and often it leads to an isomorphic version of the specification, if not the actual ruleset. These tests would be a fine set of rejection checks for the
story but the story – the ruleset for the game – still has to be there.
“Testing is not about probing for functional correctness.”
That’s what the above Acceptance Tests are for; they make sure the system does what we, and the customer, expects. And unit testing is for making sure the code works. Testing is looking for problems we don’t expect. We don’t need to learn about expected functionality – we already know about that. Testing is about finding the problems that may lead to loss of customer value. Things we *don’t* expect. If we can anticipate these things and get them included as part of our stories and acceptance tests, fine. But sometimes we need to actually see/touch parts of the working system to discover the presence and possibility of such problems. That’s testing.
And that’s what I synthesized today.
What are your views for QA integration in an Agile environment?
That was a question posed to me via email. This was my response:
Testers are part of the team. They should not be thought of as a different group and somehow apart. Testing is their primary role, but they should contribute in any way they can to help the team. That means analysis, configuration, coding, whatever. Many struggle a lot with the concept of blurring the lines between testers and programmers. I think it would be great if testers could be writing code, specifically pairing with developers, writing test code while programmers write feature code. Most places I know of don’t have testers with that expertise. Testers don’t need to be able to code to contribute as an integral part of any team.
Here’s my current view on how to do it:
- Testers should be greatly involved in the story creation process with the customers/product owners. Not only is it important that they understand what the customers want, but at this stage they should be able to point out testability of stories and help contribute to the Acceptance tests associated with the story. It is important that the tester *drives* this Acceptance test creation. It’s important for all to understand that just because they are tests doesn’t mean the tester is responsible for them! They aren’t – that’s up to the customer, or whoever is accepting the output of the team – but the tester can make sure that they are written, that they are relatively comprehensive, and that they are testable. For more information see Elizabeth Hendrickson‘s article on ATDD and Gojko Adzic‘s books on Executable Specifications
- Testers should enage frequently with programmers with what I call Dev Shadowing. Not quite pairing, the tester should sit with the programmer as they write code. That brings the testers’ skill to bear as they help the programmers with logic, tests, or anything they create. I know that many testers are apprehensive for fear of looking incompetent or resentment from programmers. I’ve found through experience that these fears are largely unfounded. Over time, as trust is built, perhaps this effort can be scaled back…as the lines blur.
- Testers, at the sprint planning stage, should determine which of the Acceptance tests, and surrounding tests, if necessary, they will implement and which the programmers will implement. If you’re using some kind of Fit-style framework, this could be a collaborative effort where the tester writes the tests and the developer writes the necessary fixtures. Or, if skilled enough, the tester could do it all. If you’re not doing something like this, the tests should be written using a unit test framework.
- Testers, again at the sprint planning stage, should be clear on which tests need to be at the UI level and which are purely related to business logic (which should exist at a mi imum in the UI). If the test doesn’t need to happen at the UI level, it is likely best handled as an automated Acceptance test (by that, I mean a test written using the unit test framework, or using a Fit-style framework). If it must be tested at the UI level, then the tester must decide if it needs to be automated or should be done manually. If you want to automate it, then hopefully you have the tools in place and the expertise and the appropriate story points/work effort assigned to account for the automation. If manual, it’s likely that BAs, and other members of the Product Owner team could assist with this effort. After all, it may just be manually running the Acceptance test as written which is up to the Customer to approve anyhow. This type of testing amounts to validation/verification and can be characterized as checks.
- Testers should engage in exploratory testing. This type of testing goes beyond the Acceptance tests, but not necessarily to their exclusion. This is the area where a trained tester can provide great value over and above the manual testing you get from other members of the team. Session-based testing I’ve found is a good method of organizing this effort so that everyone knows what areas of the system were covered in a given session.
- All tests should be run on checkin! If this is too time-consuming, at least all unit tests should be run at checkin (they should be very fast!) and all unit, automated functional and acceptance, and all automated UI tests should be run on migration to the various server environments, and perhaps nightly or on-demand. Obviously, the trick here is to make sure the tests are robust enough to not overload you with false negatives — but that’s better, imho, than the alternative.
So that’s what I would suggest in terms of integrating testers into an Agile environment. I know Lisa Crispin and Janet Gregory, respected members of the community, have published a book on this very topic.
For the last 10 years or so I’ve been a “Quality Assurance Analyst” and a “Quality Assurance Manager.” We all know now, or should, that those roles have nothing to do with quality assurance. I didn’t write feature code as a QA analyst and I didn’t dictate to programmers how to do their jobs (well, of course I did both of those things — with varying levels of success/influence — but it wasn’t part of my job description). As a QA manager I was a “resource” manager; I wasn’t responsible for teams or projects, but rather a group of testers that worked on various projects. The primary responsibility in that job description was to be aware of the availability of the people on my team through forecasting and monitoring so that when new work came up, I could assign someone to do it (and I did everything I could to make this responsibility moot).
So I was a Tester and then a Test(er) Manager. And while I did other things, my focus was always on testing – how to do it better, how to do it earlier, and how to get the most appropriate information possible to the decision makers on projects. I would not, however, characterize my current title as “Test Architect.” I’m not solely focused on testing. I’m also focused on business analysis, and programming, and sales, and support, and management and the SDLC and…well everything we do that has an affect on the quality of our work. While I attempted to have this kind of influence in my former role, I couldn’t. I didn’t have the authority, and often, the support, of others in the organization. Now, I do.
Here are some of the things I’m doing now:
- Leading an Agile transformation
- Helping the business understand internal capabilities and identify needs in the marketplace
- Helping teams coordinate efforts between local and off-shore employees
- Helping dependent project teams work together without hindering and/or blaming each other (it’s nice to finally be able to make a difference in something like this…)
- Modifying the organization structure to better facilitate small, permanent teams of cross-functional developers (analysts, testers, programmers, et. al.)
- Implementing an Executable Specification framework (with the full knowledge that they ultimately function as rejection checks)
- Integrating and aligning Security and Performance teams with project teams
I’ve been involved in one way or another on many project teams, many of which are “going Agile.” Often these teams will look to the examples of other teams inside, or outside of the organization. One such team a few years back decided to model their process against a different project that had earlier adopted Agile. On the one hand it was good, because they could tap into the project team’s processes and experiences. On the other hand, it might be bad because of the differences between them — the new project is much smaller, with different people, a different organization structure, a different user community, and a different environment. Actually, it would have been better if they came up with their own processes rather than using those from the other team.
I understand that this seems counterintuitive.
Agile teams that have been working together for a some period of time make changes. As they move through Shu-Ha-Ri they inspect and adapt; they come up with new processes to improve the workflow. That’s how it should work! But sometimes there are circumstances and contexts that limit the amount of change. Examples include org structure, upper management buy-in, off-shoring, user availability, audit concerns, and many others. If a team doesn’t have the same constraints, they would be well-served in examining any borrowed processes and practices borne out of such an environment.
Here are the chief problems I discovered when I went to help out this team.
The team isn’t together physically — One of the most important things to do on an agile project is to move all the people on the team into the same area. Communication is the key issue. It appears this team is staying in their same cube structure — testers at the end of the hall, Dev in a section over here, Management over here or in a different building, etc. It doesn’t have to be this way for this team; they can sit together.
Testers aren’t involved (yet) — I talked with the QA manager (who is trying to manage the day to day work of testers on two agile projects plus a few others…hmm) who says that the her direct reports on the project had not been invited to any analysis or design discussions. It’s still early in the project, but testers are part of the team. They need to be involved now, not just after the first build (which was due out later that week). It’s a very common pattern where each “team” – analysis, dev, test – do their bit and pass it on. And really, that’s *exactly* the type of mindset that Agile attempts to break down, but it’s one of the most difficult habits to break.
Users aren’t involved — They still don’t have any users or representatives participating in any way, save being interviewed some time ago by analysts. The other team they modeled after didn’t either, but that’s because their users were unavailable and unwilling to participate. This team’s users are close by and, from after a little questioning, quite willing to participate.
We got this team to make some changes and they continued to adapt and get better, and that’s great. Transition is difficult; moving to Agile is more than just saying “I’m going to put out a release every x weeks.” There are important aspects of real-time collaboration and constant feedback that are of crucial importance to a successful implementation. It’s a great idea to look to other teams with experience to get some ideas, but simply adopting processes without understanding context or understanding the principles behind the methodology you’re implementing will make a transition even more difficult.
Here’s the info:
Our December Agile Atlanta will feature Alex Kell discussing how to integrate testers into your Agile team.
Speaker: Alex Kell, Turner
Date: December 7th, 6:45 PM
Location: Matrix Resources, 115 Perimeter Center Place, Suite 250 (South Terraces Building – facing Dillard’s at Perimeter Mall)
Agile Testing Techniques – How to Incorporate Testers onto Agile Teams
The general premise is that “traditional” testers find it hard to make the jump from command and control phased development approaches to Agile methodologies. Furthermore, much literature suggests that development take on the burden of testing making specialized testers superfluous. This presentation talks about how testers can contribute, and can become invaluable on Agile teams.