Principles of Agile Testing
Posted by Alex Kell in Agile, Automation, Testing on June 1, 2016
A few weeks ago my company had an internal conference called “The Fiserv Leaders Conference on Testing” and I, along with some others, was tapped to do a talk on Agile and testing. We decided to write down what we feel are the principles of Agile testing (similar to the principles laid down in the Agile Manifesto). This isn’t what we ended up with, but this is my list and thus I will record it here.
Before I do, I should acknowledge this isn’t a new endeavor. Others have done this before. I particularly like this one from Karen Greaves and Sam Laing. And I’ve certainly borrowed from these ideas in coming up with this stuff below. I don’t think it breaks any ground, but this is where my head is currently at, and how I frame testing in my current organization.
Testing is a whole team activity:
Everyone is responsible for quality. When a story, or work item, is marked done, that means the team is agreeing that the work is done — the whole team, not just the testers. Everyone is accountable for it, thus everyone should be involved in testing. Anyone on the team can take on the “role” of tester, just as anyone on the team can program, or write a story. Working together is the best way to get the best quality work done.
Continuous feedback is essential:
Agile works primarily because of its built-in feedback loops that happen much more often than in waterfall or even in earlier iterative frameworks (Spiral, RUP). Scrum, for instance, has daily feedback via the daily Scrum. Then there’s feedback at every sprint boundary with the Review, Retrospective and Sprint Planning. And beyond that Release planning. When you include XP, and other common agile testing practices you create even more feedback loops on top of these. The Agile Testing Quadrants model expresses that quite well (and perhaps this does it even better). One thing to note is that “checkpoints”, “milestones” – things like that are NOT feedback. Those checks come too late (by definition). So, for example, organization mandated Control gates like sign offs, Pen testing, or Performance testing that must occur before a product is released are not valuable sources of feedback. They occur too late for us to react. Failures at these points almost always result in delays and overruns which decreases value. Check out this prezi for illustration.
Agile testing demands flexibility and the ability to respond to change:
Working on an agile team should not be like working on an assembly line. Products change, people change, environments change, organizations change, to expect one’s daily role or work to stay the same through all of that is unrealistic at best. A core principle of Agile is the ability to not only respond to change, but to be open to it. At a practical level, that means things like:
- Changing the way you write tests to suit the needs of the team.
- Adopting a new tool to solve a problem, or removing the use of a tool that’s no longer provides enough value to the team
- Taking time from the sprint to learn and practice new techniques
- Giving up some responsibilities or picking up new ones to improve the throughput of the team
Be a source of information:
Testers are not the assurers of quality (and honestly they never were). Testers provide information about how the product does, or doesn’t behave. And that is valuable! So provide that. Update stories with information about system behaviour. Be proactive with updates to the product owner. Continuously work with programmers to match up expectations with reality. Communicate with other teams and stakeholders who have questions about the system. And above all, have the courage to provide that information even if you think it won’t be well received!
Simplicity is a virtue:
When providing information, or writing tests, or executing tests, make it as simple as possible. Complexity is the enemy of information and should be avoided. When communicating about the system, don’t hide behind misleading metrics or test documentation that nobody reads. CYA is not a key tenet of Agile! When writing a test, remove useless or obfuscating information. For example, you shouldn’t need exhaustive steps – you are part of a team that knows your software. You’re no longer throwing it over a wall to a test team that has no experience or familiarity with the product. Execution should be as simple as possible. Most testing does not need to be integrated across services, products and platforms! You are usually testing a single behaviour within the confines of your system. If so, then make sure that’s exactly what you test – use test doubles (mocks, shims, stubs, etc.) as often as possible. Use tools or automation to make execution easier and faster if warranted. If automating, make it obvious what you are checking – name checks properly and make it easy to see the result matches the expectation. Design systems with testability in mind – make it easy (or at least possible!) to test the individual parts of the system.
Agile at VersionOne
Through AgileAtlanta (stewarded by Andrew Fuqua) I got another opportunity to visit a company using Agile. This time it was VersionOne, makers of a popular Agile project management tool. (Disclaimer – I’m not recommending or suggesting anyone should use an Agile project management tool. I currently use VersionOne, but I’ve also used Rally, PivotalTracker, VSTS Scrum template, Scrum for Team System, a spreadsheet, a whiteboard, and index cards over the years. All work great in given contexts.)
When you first walk into the building you see a conference room and a couple of offices. Our guide, Steve Paro, was quick to point out that there are few offices (in fact I only saw 3) and almost everyone works in team areas. Marketing and Sales are off to the left in a rather cramped series of rooms with shared “open-plan” style workstations with large visible charts on various boards, walls and dividers that appeared to split the groups in somewhat meaningful ways. Steve let us know that while all groups use some form of agile to do their work Marketing’s is closest to “pure” Scrum.
The rumpus room is the main recreational area and doubles as the location for company meetings.
A backtrack and another left turn takes us to a walkway overlooking an unexpected large open area on the ground floor. We took the stairs down to the “rumpus room” (pictured at left) which is the main recreational area and also doubles as location for company meetings. (In my time there the pool and ping-pong tables saw heavy use, which led to a key takeaway about organization culture, but more on that later.) You can’t see it in the picture, but off to the left in a little corner is a working, full kegerator.
Just past this area is where the development work is done. The group is broken up into several teams, each focusing on an aspect of the product, like Core, Analytics, and APIs. Each team has an open style work area with pairing stations. All coding is done pair-programming style and each team has at least one pair with the largest having three pairs. As people are added or switched between teams it’s always two at a time. Which leads to the question of stable teams, which is a good agile practice that the company has been adamant about maintaining.
The development area features open-style work benches with pair programming stations.
But they’ve discovered over time that not having structure around how people move between teams was causing problems. Some people felt stuck on teams and it was hurting morale. In the current model, everyone moves to another team and everyone switches pairs on a set schedule. In this way, nobody feels stuck, but at the same time there’s not large upheaval that would hurt the ability of each team to keep its established cadence. The policy isn’t interesting so much as the process in this case. The key is they had an existing policy, they observed problems related to it, so they changed the policy, adapting it over time until they found something that worked. And that in turn may change as conditions change. Here’s a question to ask about your own organization – do employees have to adjust the way they want to work to adhere to policies or are policies created to help employees work the way they want?
Another relatively recent change at VersionOne (both in the company and in the product) is their adoption of the three-tiered scaled agile model that is becoming more and more popular for enterprise adoptions. There’s a Business team that is focused on new business goals and market strategies. There’s a Product, or Feature team that focuses on defining features that will meet those goals, and then there’s the development or Delivery team that does the work. In this case, the product owners of each team work together (in fact, in the picture to the right above, the prominent work area on the right is the Product Owner team), with input from the teams, on creating features and stories for the various project teams. And this collaboration appeared to be important – the term “three amigos” was used several times. Another way they enforce collaboration is through collective code ownership – anyone can edit anyone else’s code. Shared documentation is managed using a Confluence wiki and HipChat is the tool of choice for distributed chat communications.
A view of the status of stories being worked on.
Walking through the development area and to the left is an area dominated by a giant whiteboard that doubles as a touchscreen. This is where the teams have their standups and where stories and features are managed throughout the release cycles. VersionOne has quarterly release cycles, not because that’s the only time they have product ready to release, but rather because over time based on customer feedback, they’ve come to realize that enterprises can’t consume changes very quickly (which suggests an interesting question for another day). Steve described their development methodology as ‘Frankenagile.” It’s a sort of Scrumban with elements of eXtreme Programming and their own wrinkles. Their process is managed, in true dogfooding fashion, by the VersionOne tool (it turns out they use the same release as their customers, only updating shortly before actual release). Iterations are one week, but the work flow is much more Kanban-style in that the concern isn’t that all work is done in a given iteration but rather that a sustainable pace of completed work is maintained. The key metric here is stories completed. Based on this metric they can predict with reasonable certainty how much work they can complete within a given release cycle and thus what features should be slated for a given release.
This view shows the status of builds for each project team.
The teams have three sizes for stories – 1/2, 1 and 2 with the vast majority being 1 point stories. I asked, since there is great importance on uniform-sized stories in this model, how do the teams manage to create them independently? The answer is that they’ve just naturally come to that over time. They’ve been doing it for so long that they just seem to create right-sized stories organically without any specific rules or process (planning poker cards were abandoned long ago). They do have dedicated iterations for burning down technical debt which roughly works out to about 20% of the time for a given release. Additionally, some developers are assigned to handling support calls should they arise, this duty is also rotated over time.
Continuous integration is an important part of the process. They use Jenkins to manage the builds for each team, tracking both checkin and nightly test runs which comprise both unit, and acceptance tests. They are currently iterating through the use of cucumber to create scenario tests. Steve reports that they haven’t had much success at using the tests as “living documentation” but they have seen value in the collaboration it fosters in helping refine stories.
A screen showing the high-level features planned for the next release.
I asked about UI automation, and they do have some, but mostly used for high level scenario and process flow checks. After a recent STPCon, the lone dedicated tester of the group discovered the glories of Session-Based Testing. This method of managing Exploratory Testing is quickly being adopted by the teams.
Back in the rumpus room, Ian Culling took over and gave a short presentation describing the importance of culture in the organization. When they hire, they look for craftsmen, people that care deeply about the work they produce and are intrinsically motivated to “do the right thing.” He made comment here that resonated with me about Minimally Marketable Features (MMFs). He said they don’t equate to “just getting the job done,” they have to be right; these are features upon which your software is judged by users. It’s a key point about how sometimes commonly used terms can take the focus away from what really needs to be done.
Ian Culling talks about the importance of culture at VersionOne.
They also look for anti-establishment types that won’t be readily cowed by corporate edict. He told a great story about guerrilla coffee makers that I won’t repeat here, but suffice to say employees are empowered to change their environment, even if it involves company money, to make their work more effective. To judge whether an individual will fit well in the organization, a candidate actually works with the team for up to a day, and anyone on the team can veto the hire. Earlier I commented about the recreational equipment — if you have these things does anyone/everyone use them? If not, why not? Is it because they fear to use them because of judgement from management, or even other coworkers? If these questions make you the least bit defensive, you might have a culture issue.
Overall they strive to create a culture of reflection and experimentation. They of course encourage regular retrospectives on the teams, but also foster experimentation in other ways. One is by sponsoring a “hack week” where employees work on whatever they want. Some of the features in the product, including the new “Team Room” feature, are a direct result of this initiative. They also have a Makerbot.
Agile at Daxko
Posted by Alex Kell in Agile, Automation, Management, Testing on November 15, 2012
I had the opportunity to visit Daxko through Agile Atlanta and their Agile Tours. I had already heard of the company and their foray into agile through Claire Moss who is, by the way, an excellent world-renowned tester and local Atlantan. That’s why when I heard about the Tour, I knew I had to sign up.
The team we were introduced to was fairly typical. It’s a small team with a couple of programmers, a tester, and a UX designer. Additionally they have a design lead and a development lead, who along with the designer formed a Product Owner team. Now it’s not unusual to see a PO team these days, but I did think it interesting to see one on such a small team. I think it’s common for teams to be burdened a bit on “grooming” process; they’d rather be sprinting on current work than thinking about future plans. On the other hand, leaving the entire grooming process up to a single PO can be overwhelming.
The product owner team refines ideas until they’re ready to be worked on as stories for a given sprint.
This team appears very committed to customer experience. The fact that they have a dedicated UX designer who makes it a point to speak with and visit with customers on a regular basis is a testament to that. The designer showed how she spends time (along with input from the team) making exemplars of their customers through the use of personas. She was very quick to point out that these personas are evolving and can’t, by their very nature, be viewed as explicit representations of what all their customers are like, but it’s a helpful guide for what types of features and the type of presentation they should be shooting for in their products. This analysis along with frequent user-testing of concepts helps create ideas that the product owner team then grooms into actual stories that the team will work on. (Another point driven home by the designer was that she no longer has to create wireframes of interfaces ahead of time. If she wants to prove out a design, she creates it right within a browser. When working on stories, she can pair with developers and build out “real” UI components and shape workflows through system within the sprint! Shocking!) The image to the left shows their epic board (my term). They take these large, and somewhat unproven concepts and groom them, with increasing involvement from the team as they move to the right, until they are ready to be worked on within a sprint.
The team uses story mapping to help plan out their stories and schedule.
To determine what stories to work on and when, they use Story Mapping. In fact, they brought in Jeff Patton to help show them how to do it. The image on the right shows an example. That top row of orange cards represents the actions their users take, in the general order they are performed. This again is learned from the research done directly with customers. Then, the most essential parts are written on the yellow cards. That first row of yellow cards forms what’s commonly known as a “walking skeleton” or the basic functionality you’d need to provide business value. Then the team continued to add additional functionality further down on the map; the lower the importance, the lower they appear. The blue tape shows how the pieces of functionality should be released as a collection, so this board shows that there are likely to be three planned pushes of functionality.
A list of planned releases of functionality color-coded by confidence.
Which leads us to the Roadmap depicted to the left. It shows a listing of planned dates and the content that will be delivered on those dates. Note the color-coding. The items in green are dates they are fairly certain they can meet while as things go out further, the confidence decreases. Due to the nature of their business, making firm commitments even 5 months out is a dicey proposition since market needs and priorities can change quickly. Thus, even though the team feels pretty confident they could hit a date that far in the future, it doesn’t make sense to commit to it until they know more information. Seems like a decent implementation of Real Options to me. I got the impression that the rest of the company isn’t completely enamored of this practice. I can imagine that there may be some larger efforts that absolutely need to be delivered by a certain date for business reasons (say, fiscal year end, for instance). In those circumstances it’s not a question of when you deliver, but what. Now that’s a relatively basic agile concept of changing scope but fixing date. If everything is flexible, well, that’s great and I’ll be interested to see if they continue to have success doing things this way.
A basic Kanban-style board with some interesting wrinkles.
And that’s where we transition from the Product Owner team to the rest of the team, what they term the Product Team. On the right you can see how they track their progress. Much of it should be fairly obvious to anyone with experience with Kanban boards. the Green stickies on the left represent the stories, the yellow stickies represent tasks, the red stickies represent defects and the blue stickies represent acceptance tests. The yellow stickies get moved to the right as they go from “In Progress” to “Done.” There are a couple of differences from most boards that I’ve seen that make things interesting. First, note the little magnets connected to some of the tasks. Those represent team members. Each person gets three and you can have no more that on the board at any time. This keeps their WIP limit under control. Genius! Second, do you see those darts on the upper left of the image? They are used to target a note to let people know that this is something someone is stuck on and they need help. Basically it’s an indication that the team should treat this with urgency and collaborate on a response ahead of anything else. Third, note that there is a division a little past halfway on the board. The Acceptance tests are tracked as specific items on the board. Only if those acceptance tests are all in the Pass column can a story be considered done (even if all the tasks are in the Done column). And really, this is the true indication that you’ve completed some work – does it do what we need it to do? There is a growing trend of folks tracking acceptance tests rather than stories/tasks as the true indication of whether work is Done. I like this idea quite a bit. One of the product owners described it as the difference between Output and Outcome. You need to have output from the team, but the reason you do it is to provide the desired outcome, ultimately, that is what you want to track. Finally, look down at the left bottom of the picture. You see a set of stories that are “on deck.” These are groomed and ready to be worked on. If work gets done early in the sprint, or even if priorities change, one or more of these can easily be added to the board. This is another basic agile rule-of-thumb that you should have at least 2 sprints of work groomed and ready at all times.
Big visible information is available all throughout the Daxko team room.
It’s critical to be absolutely clear on what it means to your team to be “Done” with a story. I like how this team has not only defined this, but made it into a big sign and visible to anyone around. It’s also evident that this is not boilerplate but specific to this team (see the mention of browser types and resolutions) and that it’s evolving. The team was very explicit that they understand that any process or practice they have is only in use as long as it provides value to the team. If they deem it isn’t, they change it or remove it. One example is the burnup chart to the left. They mentioned that this was the first sprint that they were tracking in this way. I pointed out that a flat line for the first four days of the sprint was of dubious value. Perhaps smaller stories would be in order to lower the risk of not delivering anything in a sprint. Additionally, they’re also just starting to track cycle-time. They define it as the time they begin work on a story to the time it’s available for customer use (they’re also tracking the time at which a story is done). I’ll be interested to find out how that information will help them in the future. And speaking of completing stories, in the bottom left of the picture you can see the metal frame of a large bell. They ring that bell whenever a story is complete. There’s always room for a nice Pavlovian reward if it helps the team’s morale!
After each sprint, the team has a retrospective. They showed off a whiteboard from the last session that used the metaphor of the team as a ship with wind driving the team forward and anchors weighing the team down. Hey, whatever floats your boat! The ScrumMaster said that he changes up the metaphors and retro techniques to help keep things lively and fresh. The team also does demonstrations of their work to the greater company. Every other Tuesday, this team, and the other teams across the company, have a video conference where each team gets a period of time to show off their work. I’ve noticed that the demo is one of the first things that agile teams drop from the standard Scrum ritual list and I’m glad to see the practice alive and well here.
After release, the team also monitors their applications. To do this, they use services from New Relic. It provides the team with a real-time view into what their customers are experiencing. The information presented was impressive and looked very nice. If you can get away with a third party monitoring your application performance, it’s certainly something to look into.
While the team certainly seems happy and successful, they are continually looking for ways to improve. They posted a list of some challenge areas right on the wall. One is the challenge of distributed team members. They recently hired a remote programmer and while they do store story information and progress electronically, that individual doesn’t get the benefit of the environment and the physical board and charts the team uses. They’ve been experimenting with a controllable camera, but it appears the jury is still out on that. I suggested they check out the work of Joe Moore from Pivotal Labs who writes a lot about tools and techniques for integrating remote team members.
I certainly would like to thank Andrew Fuqua (@andrewmfuqua) and Claire and the rest of the Daxko team for their openness and willingness to share their processes and ideas. I wish that I had been able to stay longer and discuss more things…perhaps next time.
Levels of Feedback in the Agile Enterprise
Recently I gave a lightning talk at Agile Atlanta on this subject. Here’s the Prezi. Feel free to click through that as you read along to this post!
The idea is that there are several types of feedback that we get on a project (I discuss three) and they occur at different levels – those levels are related to the Scaled Agile Framework from Dean Leffingwell and others that have either independently conceived it, or have built upon it. But we’ll get back to that.
The first type of feedback is project related and it’s facilitated by the Agile process. The thesis here is that Agile and other “traditional” SDLC methodologies are all guiding the same type of work: we have requirements of some sort, we write code, we test it, etc. What Agile does, intentionally, is create frequent feedback points. Scrum, for instance, talks about daily scrums, sprint planning, release planning, sprint reviews, demos, retrospectives – all of these are specific, and prescribed, points of feedback that are built into the process. There’s nothing about a waterfall process that says you can’t inject these points of feedback, it’s just that they’re not built-in.
The second type of feedback I discuss is Testing. For this I trot out the Agile Testing Quadrants originated by Brian Marick and built upon by Lisa Crispin and Janet Gregory. There are plenty of sources out there to get more information about the model, so I won’t do that here. The point is that testing itself deliberately provides feedback, and the different approaches and techniques represented in the model give you a variety and a frequency of feedback not, again, prescribed in traditional methods.
When most people talk about Agile, they’re referring to happenings at the team-level; they talk about the meetings, and perhaps XP practices. There are other levels in the Enterprise, and those are providing feedback too. In the typical model, there’s the team level, the product level, and the portfolio level. Agile process is happening at all these levels, as is testing. The frequency and types of both changes, but it happens and it’s critical.
There is a third type of feedback highlighted here, and it involves the idea of “Control Gates.” Control Gates are those steps in the process that are demanded by policy, laws, or some form of structural practicality. If there is a failure at any of these points, the work that happened before is rejected and must be resubmitted. The thesis at this point is that this is not feedback. It comes too late to provide value over and above the amount of re-work it incurs.
The most important aspect of the discussion involves moving aspects of “Control Gates” sooner in the process. For instance, if we do Pen Testing last, after all other aspects of the product are completed, we’re not getting valuable feedback. It’s too late for that. If that test fails, all progress grinds to a halt and we have to go back and fix old code – and then go through the whole deployment process again. A better option is to treat these control gates as a formality – not as a source of feedback. We should be able to run the same tests for every build if we want, every check-in possibly. By the time we get to that control gate, we should have run the same tests hundreds of times already.
This is true for any type of control gate. For example, lots of projects have a formal Support Transition process involving training and documentation and sign-offs. Instead, it would make more sense to have a support representative actually on the team, learning along with the team about the project. By the time the hand-off occurs, all the learning has happened and it’s a smooth and fast transition. It would be too late at that point, for example, to learn that Support had no means of monitoring or troubleshooting the application.
Failing Fast
Posted by Alex Kell in Agile, Automation, Management, Testing on April 16, 2012
My company has a Director of Innovation. Joe is, predictably, a pretty inspiring guy and often sends out emails with interesting thoughts and ideas to spur innovation. Recently he sent out a message talking about the benefits of Failing Fast. In it he referenced the Gossamer Condor, the first human-powered aircraft capable of flying in a controlled fashion for more than a mile. Paul MacCready won the contest in 1977, 20 years after the contest began. Why did it take so long? Well, the biggest challenge was that failure was too costly. The designs were often good, but cycles of assembling the materials, testing, repairing and retrying could take as long as a year. As the attempts wore on, it became more and more difficult to get them financed.
MacCready took a different approach. He knew that no matter how good his designs, the likely result would be failure, just like all the others before him. As a result, he expected failure and built his prototypes with that in mind. He use common and cheap materials that were easy to work with and easy to repair and replace. As a result he could test new designs in a matter of weeks rather than years.
So how do we apply a fail-first thinking in software development? Here are some approaches, techniques and models that are borne of this philosophy that can have effects at all levels of your organization:
Team Level:
Test Driven Development (TDD) –
This is an approach used by programmers at the lowest level of software development. The idea here is you write a test that will initially fail because the code hasn’t been written yet. Then you write code until it passes. Certainly this helps you fail fast, in fact, it encourages you to fail first! The true power, though, is in the freedom it gives developers to try things they otherwise wouldn’t for fear of failure. With a battery of tests already created, the programmer is free to experiment and try out more innovative designs because they know they have that safety net. Failures are reported instantaneously, as are successes.
Executable Specifications –
With this approach, analysts and designers are called upon to work with programmers and testers to create requirements which are written with examples that can be hooked to the code. When the code is written to do what the business wants, these examples pass. If it isn’t, the tests won’t pass. Now that’s fast failure. In the past, such problems (programmers and testers misunderstanding requirements) might not have gotten caught until much later in the process. Of course, all this presumes the design or feature was correct.
If you’ve been doing this all along, a new feature might be proposed; the tests are written and the feature is coded and the tests pass, however an older test fails. Just as in TDD, you now have a safety net of “living” documentation. Before you had to pore through the requirements documents to find the inconsistencies; now all you need to do is run them. Additionally, if done properly, these specifications become the documentation. There’s no worry about them becoming out of date or inaccurate with respect to the code because they are directly tied to the code.
Project level:
Short Iterations –
This is a tenet of Agile methodologies. Teams complete releasable units of prioritized work in a short and predictable amount of time. If we need to change course because of some unexpected event, we don’t have to scrap the foundations of work that won’t be complete for months, we only have to scrap, at most the amount of work related to the length of the iteration (or, for you Kanban folks, the cycle-time for the largest unit of work in process).
Small, more frequent releases –
In some situations, there are restrictions on how frequent and how small we can make our releases due to overhead (training, deployment, regulatory testing) but we should be working on making these as small as possible. Just like with iterations, if we make the wrong call on a feature, we can react faster – instead of 6 months to a year to implement a change, we can potentially do it in an iteration’s time, with no disruption to the business as would be the case with emergency hotfixes/patches, etc.
Business level –
Feature flow and prioritization –
Too often businesses decide on goals unique to each business unit and then those units create projects independently to meet those goals. Often these projects compete with each other for resources and capital when only parts of them are actually directed at the business goals. We shouldn’t be focused on the projects, but rather the features in those projects that are most important. Models exist that show how to link these goals to prioritized features. The business units are now not concerned with independent projects, but rather they are working on those aspects of the features that they have the capability to produce. The business units can then speak in terms of continuous delivery of features within products rather than entire projects released all at once. This keeps us focused on what we believe are the most important things given the best information we have at that time. If it’s wrong, we can adjust our backlog of features based on the new priorities. The alternative is to release updates based on information from a year or more ago and to cancel projects with nothing to show for them.
The Embedded Tester
Posted by Alex Kell in Agile, Management, Testing on April 12, 2012
I hear it a lot at conferences, work meetings, interviews, discussions with managers, etc., the idea of “embedded testers.” The utterances are along the lines of, “Our testers are embedded on development teams!” or “How do we embed our testers with our developers?” It unfailingly puts me in the mind of a reporter from say, CNN wearing an ill-fitting helmet and flak jacket*, standing there as well-drilled troops go about their business. These journalists are working alongside the military, sometimes getting into actual firefights and other dangerous situations, but they’re not part of the team. They are most certainly outsiders with a different agenda (although it doesn’t always work out that way – more on that later).
When we talk about embedding testers – when we use that language – we’re implying that we’re taking a member of a separate Quality Assurance group and dropping them into a team of programmers, much like dropping a reporter into a military unit and sending them to the front line. No wonder testers are apprehensive and no wonder developers are resentful. Just using the phrase suggests a cargo-cult mentality is at work; we show a misunderstanding of the reasons for doing it and whether they’ll even apply in our particular context. And as a manager, when you use this phrase, you’re thinking in the back (or perhaps the front) of your mind that, just like a news organization, you can simply pull that person out of the team and back into the testing pool and the team will be unaffected.
In Agile, testers aren’t “embedded” on teams any more than programmers are, or analysts, or any other role that is needed on the team. To say that they are suggests that this is an option, or a particular strategy you might employ to help with Agile development. It’s not! It’s an essential part of it!
And no, I’m not saying that all testing has to be done by the team. There are reasons why you might want, or be legally obligated, to have independent testers outside of the team. Additionally, you’ll want your users involved in evaluating your work.
So above I mentioned I would talk more about the “different agenda” of embedded journalists. The social science is pretty new on all of this, but it turns out that there’s a bit of Stockholm Syndrome involved. These people are together under quite stressful situations, and often, in spite of their roles as impartial providers of information, they become overly sympathetic to the point that they glorify the actions of the troops and neglect, or leave out truths related to the enemy combatants. In fact, administrations have counted on this, encouraging embedding in order to provide more support for the war in Iraq.
“Aha!” you say. “This is exactly what managers are afraid of! Testers fraternizing too closely with programmers to the point of hiding quality problems from management! You need that adversarial relationship between devs and testers!” Ok, stop. First of all, shame on you for drawing parallels between software development and war! (ahem) And Second, I *knew* you didn’t really want testers as part of the team! (pwned) Look, Agile teams are perfectly capable of testing their own stuff and pitting testers against programmers is a fast track for making testing irrelevant. You’ll need to establish trust and choose metrics that reflect the goal of the team. Then you’ll have a good team, and not just a bunch of people embedded together.
* It turns out that these days the military won’t spring for such gear – the reporter is responsible for their own. (Then again, maybe that contributes to the lack of fit.) However, it’s not like the military is saving much by doing this.
Fixing the On-Ramp
Posted by Alex Kell in Agile, Management on March 13, 2012
I figured it might be fun to continue this one-sided discussion about Productivity, so here we go! For lots of managers, there’s a holy grail – the idea of 100% efficiency – which, as we all know from the 2nd Law of Thermodynamics is impossible! (What? That’s just for heat engines? oh.) No, the real reason it’s impossible is because it’s unsustainable. At some point, variation will enter the system which will cause disruptions to that 100% efficiency. If you’ve ever done the “airplane game” at some agile training workshop or other you’ll know the concept. If not, think of a traffic jam. If a highway is full of cars, adding more cars won’t speed up the traffic; in fact, it’s the opposite. Adding more cars will slow it down even more.
For example, take a machine that can stamp out 60 widgets in a minute. It takes three inputs that must be entered into the system simultaneously. If any one of those inputs varies by a little bit, the machine will fail and the new inputs will start piling up behind. And then you have to stop the line, and fix whatever the problem is. Maybe it’s a quick fix, seconds even, but you’ve messed up the efficiency and you’ve created defects.
The fix is to introduce buffers into the system. By slowing the system down, problems caused by normal levels of variance will not reduce the efficiency of the system. This is kind of the same idea that created those ramp meters at the highway on-ramps.
So how does this relate to Software Development? If we try to complete more work in a given timebox than the team is capable of completing, the effort expended in trying to complete that work will actually make the team less productive, not more. Putting an amount of work that is lower than (imagined) capacity will increase the likelihood that it all gets done regardless of normal levels of variance, creating a more predictable and stable cadence. This will be more efficient than stuffing the iteration with as much work as possible. This is why Kanban has WIP limits. It creates that buffer to help ward off the effects of variance in the system.
How to Measure Productivity
Posted by Alex Kell in Agile, Management on February 27, 2012
I hold a weekly meeting to discuss all things quality and iterative development. This week, we tried to define Productivity. We came to the conclusion that the way we want to measure productivity in the software world is different than in the manufacturing world. People aren’t machines or even assembly lines. Simply being “up and running” doesn’t mean we are being productive. One common example is the manager walking around to see if people are surfing the Internet instead of being “productive.” Someone shared an anecdote of a dishwasher that finished early, but rather than incur the wrath of the boss for sitting around doing nothing decided to dirty up the dishes and wash them again. That’s not the kind of behavior we want to encourage.
We discussed why using # of Story Points/Ideal Days (i.e. Velocity) to measure productivity is a bad idea:
- It’s not fungible , i.e. points aren’t transferrable between teams or across projects. So if team A has a 20 point velocity and team B has a 40 point velocity it doesn’t mean B is 2x productive as A. In fact, it doesn’t even mean they are *more* productive.
- Once you start rating a team on productivity based on points, guess what? The team will start showing a dramatic increase in Velocity!
So what’s a better measure? You could simple count the number of actual stories you output over time. That could work, but it’s also susceptible to gaming. (Although, something that encourages lowering story size is probably a good feedback loop to employ!) Cycle time is also a metric you might use. This measures the time from when a story gets into the iteration to when it is marked Done. If your average cycle time is longer than a normal iteration, well that’s an indication of low productivity!
None of these are perfect. That’s why it’s also a bad idea to rely on one metric. Use a combination of tools to focus on areas of concern and improve them. If velocity is going up, but cycle time is staying the same, there may be some problems with story estimation – maybe one voice has started to dominate the process. If output is up, but velocity is down, there may be a problem with prioritization (we’re only working on easy stories).
We also discussed Throughput as a concept, being the result of Sales – Cost of Raw Materials and then Productivity = Throughput/Operating Costs. This comes from Throughput Accounting and it isn’t often used in a software context. I think it’s fine to couch Productivity in terms of $; it’s certainly fungible. However, again, software generally isn’t sold in units that can be created many times a day! You’ll need to figure out productivity during the project in some fashion before it even gets put into active use. Still, this is all the more reason to have small features that can be put into use as soon as possible.
Contract and Integration Service Testing in a Componentized Enterprise Model
Posted by Alex Kell in Agile, Automation, Testing on February 23, 2012
(Presented at Agile Atlanta on May 1st, 2012 – Prezi)
I’ve been working a lot over the past six months on creating an automation framework for our service development teams. In our relatively large organization, we have have teams devoted to different horizontal strata of software development. So, for good or ill, we have database teams, mainframe/API teams, service layer teams, and UI teams. If you’re at that service layer, and that’s all you do, you should be taking a different approach to testing than if you had to do all of the testing for vertical slices of product functionality. With that in mind, here’s a proposal for a presentation on the strategies and the automation framework we’ve come up with. I welcome any feedback or thoughts!
————————————————————–
How do we ensure appropriate testing and feedback for large Agile Enterprise projects where the value delivered may not be end-user facing features? Such is the case for many enterprises working to establish a Service Oriented Architecture. A lot of knowledge exists for agile teams delivering customer-facing products. Much of that work focuses on acceptance tests being written in clear business language that is understandable by the whole team and stakeholders. This approach becomes more complicated for component delivery teams developing Shared Services that are often seen in large Enterprise development organizations.
Services can be tested in a number of ways and the approach that’s best can vary greatly based on the context of the application and environment. The presentation discusses common patterns of testing services and focuses on an acceptance test solution appropriate for teams working on the shared service layer within a componentized Enterprise model. The solution is based on a “universal” fixture that works for any type of object-based service and is accessed via table-based acceptance tests in FitNesse.
Process/Mechanics
The talk will be of a standard presentation format with slides and discussion. The first part of the presentation provides background on testing in an Agile Enterprise model and the various approaches to Service testing. (15) The second part of the presentation discusses the creation of specific framework to aid in service contract verification that is helpful for specialized teams working in a componentized Enterprise development model. (15) The last part gives examples on basic usage and on how the framework is used in a real-world environment. (30)
Learning outcomes
Knowledge of how Agile in the Enterprise affects our approaches to testing and feedback
Understanding of various methods of service testing and how to choose what’s right for your team
Details and examples of how to employ a framework for testing services appropriate for the componentized Enterprise model.
Agile Allure
Posted by Alex Kell in Agile, Automation, Management, Testing on December 12, 2011
Through the wonderful Agile Atlanta, I got to visit another company’s location this week – Allure Global. They make dynamic signage for movie theaters – concessions and ticketing, etc. They are an Extreme Programming shop, which means they use the XP practices like TDD, collective code ownership, and others. They have two week iterations. They code in Java and use Jira as their project management tool. I took a couple of pictures.
This first picture of the workspace for one of their delivery teams.
Note the monitors side by side at each station. That’s because they do pair programming, another of the XP practices. When they come in to work, the team members pair up and start working on the days tasks. They don’t keep the same pairs and anyone can sit at any workstation – all of the code is shared and can be worked on by anyone. Checkins are signed by both members of the pair.
The second picture shows their dashboard and the status of their continuous integration.
They use Jenkins to manage the automatic build, test and deployment of code. The bottom screen is green because the build for that evening was successful. You can also see the number of tests (or checks) that were run. Those are the tests that run as part of the build and they run FAST. All 20 some-odd-thousand of those tests run in a matter of seconds. These tests use mocks and other test doubles which allow them to run fast, but also to check only the specific code addressed. Having a fast-running, comprehensive suite like that makes it so much easier to refactor and redesign because you’ll get immediate feedback if you’ve broken anything else in the process. It’s a great feeling!
The top screen shows the status of the automated functional and/or Acceptance tests – see zone two of the testing quadrants! These tests are more integrated and (as a result) slow and work at the story/feature level and are of more interest to the customer/product owner.
The top screen also rotates to other information radiators important to the team, like a burndown chart.
Much thanks to the friendly folks at Allure Global for sharing their environment and processes with me!
Manage to Test 