Agile Testing – Integrating the QA Team into Iterations

Let’s start with a simple (and likely accurate) assumption – you have a QA team.  This may not be true, in which case you can just skip ahead to the recommendations below on how to move forward.  However, for those that are still with me, I have a few other suspicions about your QA team:

• the group is led by a senior QA lead or manager
• the QA testers all report to the QA manager (both as HR report and for routine work direction)
• the QA team was added to the organization after some traumatic experience (may not be true, but if it is this is the source of great drama!)
• you have a 4:1 or 5:1 developer:tester ratio (or worse)
• the QA team is separate from development and may even report to the VP Product Development (or higher) as an independent organization
• the QA team pushes hard to get requirements and design documented up front so that they can build their test cases
• QA is most involved in the final stage of integration and regression testing as they seek to confirm that all features of the product are functioning as expected
• QA is repeatably frustrated that their suggestions for improving the product are ignored
• developers are not allowed or expected to be involved in testing
• QA is perceived as the bottleneck that prevents the product from shipping
• the QA manager has a say in making the ship decision, but readiness reviews with senior management are strained as the development organization is afraid it will be surprised by what is communicated by QA

The reason these suspicions are likely true is that this is the textbook canonical form for how to organize a software development organization.  It’s also common for the simple reason that far too many testers have had their integrity compromised by project managers or developers that try to bully them into shipping a product before it is ready.  I would even go so far as to say that most process management disciplines (such as CMM (Capability Maturity Model), ISO 9000, or DFSS (Design for Six Sigma)) intentionally set out to force such a structure and assert that the Quality organization is distinct from the rest of the operational entities and must have sponsorship within the executive team.

Well…”Mr. Gorbachev, tear down this wall!”  You may have the best of intentions behind this structure, but it is very likely wasting time, creating even more frustration and animosity between the teams, and delaying product releases without appreciable improvement in quality.

Cranberry Bog - Warrens, WI

In earlier posts I described my preferred structure (see Scaling the Agile Development Organization), but it’s worth some extra effort here to explain in more detail how all aspects of testing should be handled.  In particular, we also need to explain how we move from a structure and role expectations described above into this new model.  Since responsibilities shift and required skill sets change, this transition is not natural and will require explanation, training, and coaching.

In the end state, there are three distinct forms of testing in use:

• Unit testing – created by the developers and exercised on a regular basis by the continuous integration process
• Feature testing – created and executed as part of each iteration by the manual testers assigned to the feature teams
• Regression testing – created by your automation testers and executed at least weekly

I know what you are thinking – we already have this, so we’re done!  Possibly, but I doubt it.  I am guessing that while your teams might say they do these things, that once I describe them in more detail below you will have to admit that what they are doing is different both in structure and degree.

. . .

Unit Testing

Unit tests are written by another developer as part of pair programming or by the developer that built the objects under test.  The goal is to design an architecture that allows most of the objects and encapsulated business logic to be tested with each build.  You are measuring the code coverage delivered by these tests and you have stated goals to maintain or improve this coverage over time.  Tests are meaningful and involved and exercise edge conditions that are not likely to be tested or accessible through other means.

. . .

Feature Testing

Testers have been assigned to each feature team with at least a 4:1 ratio within the feature team and they sit with their associated team.  The tester is involved in the iteration kickoff and spends the first day or two of the iteration writing exit criteria for each of the stories.  These exit criteria drive the design of the test cases, most of which are written in parallel with the development of the feature as the tester is directly involved in the ongoing informal discussions and daily scrum.  The feature team develops its stories one or two at a time instead of having many in parallel so that the workload of the tester is smoothed out throughout the entire iteration and there is not an impossible workload on the last day of the iteration.  Developers routinely help with testing the features built by others if the tester is overloaded – this happens naturally because the team is self directed and they all want to get the feature done so they can move on to the next feature.

. . .

Regression Testing

There are one or more testers responsible for owning and updating the automated regression suite.  The regression suite has been carefully designed to ensure that it exercises all important interactions within the system; it is NOT just a repeat of the functional test cases done during feature testing but is likely a subset of that combined with use cases that span features.  The automation testers meet with the feature team testers on a regular basis to understand new features that have been built and how they should be tested.  Updates to the regression tests likely lag behind new feature development by one or two iterations, but in general most new features have been automated before the product needs to ship.  The automated regression tests are run at least once a week.  These tests represent the majority of the testing that will occur during “stabilization” or “hardening” after the code freeze has occurred, and stabilization is complete within a week or so of the end of the last new feature iteration in the release.

. . .

If your QA team is engaged in these test activities as described, then they split between one of the two activities described above.  It’s difficult to achieve the goals of the automated regression testing described here without using someone very skilled in the art (see Automation and the Three Bears – Scene 3: Just Right).  If you do have that person, then they may be able to teach one of your current team members how to develop automated tests within their framework or else use them to execute and interpret the automated test results.  This is an ongoing process with new defects being constantly logged, fixed, and confirmed.

The remainder of the team should be assigned out into each of the feature teams and will receive their day to day task management from the scrum master in that team.  Their activities include product design, test planning including writing exit criteria and test cases, and executing manual tests.  They are responsible for ensuring that their feature meets the functional requirements originally communicated in the story by the product owner.

Cranberry Bog - Warrens, WI

What does the QA manager do?  Well, if this is all working as expected, then the manager has little by way of day to day responsibilities outside of their HR role.  For this to work, they MUST let go of being involved in the details of the daily work within the teams.  They are there to help and coach, review work product, and make sure that the testers are delivering on their role as expected, but they are chickens, not pigs.  In the end, they make their decision on whether the product is ready for release based on the advice of the features teams and not because they have tested the product with their own hands.  I have had great success involving the QA manager in an operational role handling support escalations – each of these represents a quality or process breakdown where the defect needs to be fixed and the process also examined to learn from the mistake.  While it sounds like this involves them in tactical break-fix work, I believe it is the exact opposite – there is nothing more strategic than creating a “learning organization”.

Once you have achieved this structure and you have everyone working effectively in their new assignments, there are a few virtuous improvements you will notice within your QA team and across development as a whole:

• the workload on test has been smoothed – there are no dramatic peaks at the end of the iteration of release cycles
• stabilization happens much faster – there is less testing done and it is done more efficiently through automation
• unit testing forces design and test activities that result in higher quality product being created by the developers
• testers have the opportunity to influence the feature design as it happens and find that their ideas for improvement are being used
• developers understand their role in ensuring quality and are involved during iterations and help out with stabilization if needed (although this should be minimal due to automation)
• the quality of the product is transparent to all and ship decisions become routine
• no one worries about the independence of the testers because they are empowered within their team to do the right thing

Let me be the first to admit that these changes will be seen as threatening by all involved.  Not everyone will agree with the goals, and some may decide that they don’t like their new responsibilities.  However, I have seen several organizations make this shift successfully and I know that you can do it as well.

Advertisement