The Quality Problem Began Before Testing

At a research and advisory firm, the internal acceptance tester was finding defects faster than she could log them. More testing wasn't the answer. Each team wrote down, and took ownership of, what done had to mean before work left its hands.

02 SCOPE
The condition

Every release reached acceptance testing with defects that had already passed the teams' own tests.

Passing their own tests, failing hers

At a global research and advisory firm, one person ran user acceptance testing for products that several delivery teams built. Officially she was an internal customer, since the products were used inside the firm. In practice her role was no different from a paying customer's. She used what the teams built and told them whether it worked.

Lately she had been telling them it didn't. When we joined her and the leader of the delivery teams on a call, she was finding defects faster than she could write them up as tickets. She didn't need a dashboard to explain what was wrong. Her example was a search screen that showed a small fraction of the records she could find by checking manually. The developers had told her the search worked. Technically it did return data. Returning data is not the same thing as working.

We asked how often this happened. Every release, she said. Sometimes it was minor, like a wrong label or an inconsistent date format. Sometimes it was a core function. The teams were passing their own tests, and the product broke the moment she touched it.

A loop that fed itself

Every piece of work she rejected went back into the backlog as a defect fix. Those fixes used up capacity that had been planned for new features. The features that didn't get built rolled into the next release, which arrived with its own defects. Month after month, the quality debt grew faster than the teams could pay it down.

The teams felt it. She told us they were tired and frustrated, and that some team leads had started floating shortcuts out loud: skip acceptance testing, release without a test plan because the deadline was tight, write the documentation later. In the moment, each shortcut sounded reasonable. None of them were. She understood why people reached for them. They were drowning, and a shortcut felt like the only way to get some air.

We asked one more question. How much of the backlog was now defect work rather than new features? Almost half, she said. We called that technical bankruptcy. Shipping early with some defects is like taking out a loan, and that's fine if it creates value now and can be paid back later. It becomes bankruptcy when the organization spends more energy working around its systems than improving them. She said at least three of her products were already there, and that buying a replacement had started to look easier than fixing them.

Figure 1
How rejected work fed the next release
  1. Release reaches acceptance Teams' own tests passed
  2. Where it broke Tester rejects work Defects found on first use
  3. Where it broke Fixes take feature capacity Returned as defect work
  4. Where it broke Features roll forward Into a crowded next release
  5. Where it broke Next release, new defects The loop starts again
Each rejected item took capacity away from planned work, so the next release arrived late and carrying defects of its own.
Figure 2
What the backlog had become
  • About halfDefect fixes
  • About halfNew features
Approximate, from the tester's estimate
The tester's own estimate. When defect work grows to about half the backlog, fixing yesterday starts crowding out building tomorrow.

Three teams, three definitions of finished

Acceptance testing didn't create the defects. That was just where they became visible. The real problem was earlier. Neither the organization nor the teams had ever agreed on what quality meant before work reached the tester. Every team applied its own standards, its own way of validating, and its own judgment about when something was finished. So teams believed work was done, and the tester kept finding problems that should have been caught much sooner.

Putting the three teams she worked with side by side made the pattern clear. The first team already had mature quality practices. It released monthly, validated thoroughly, and brought business stakeholders in before work reached acceptance. Its work rarely bounced back. The second team had no formal definition of done, validated inconsistently, rarely demonstrated finished work, and often pushed partly validated functionality into acceptance. Most of the tester's defects came from this team. The third team was in between. It had quality processes but didn't follow them consistently, and its defect levels matched.

Quality wasn't random. It reflected the standard each team applied before calling work complete. Once the delivery leader saw the comparison, the case for a shared definition of done was obvious.

Figure 3
Same tester, three standards
Definition of doneWork shown earlyDefects at acceptance
Team onePracticedYes, to stakeholdersRare
Team twoNone formalRarelyMost of them
Team threeInconsistentSometimesIn between
Defects reaching acceptance followed the discipline each team applied before calling work done.

A checklist each team wrote and owned

Over the next several weeks, we worked with each of those teams to formalize its definition of done. Each team built a checklist of what finished work had to meet before it could be called done.

The details varied from team to team, and that was intentional, because the teams did different work. The goal was the same for all of them: a quality standard that everyone could understand, check work against, and apply the same way every time. The checklist was the team's own. The teams wrote it, used it as part of their normal delivery rhythm, and shared it with the people who depended on their work.

That's an important difference from what the organization might have done instead. Adding more testing at the end would have caught more defects later, at the most expensive point. A team-owned definition of done moved the check to the moment a person decided the work was finished.

A delivery team gathered around a monitor, working through software together.

From whether it works to whether it is what we wanted

A few weeks after the new standards took effect, we met with the tester and the delivery leader again. She said the bounce-back rate was down. She deliberately didn't give a number, because it had only been a few weeks and she didn't want to overstate it. But the difference was visible. Work was arriving more complete, and the defects she still found were smaller. The broken search from before would never have reached her. The team would have caught it.

The delivery leader noticed a different change. Team leads talked about quality differently, and the shortcuts they had been justifying a few months earlier had stopped coming up. In reviews, teams were demonstrating working functionality instead of describing what they had built. The tester was being invited into those reviews instead of getting releases she had never seen until they landed in her queue.

The tester's closing observation captured the shift. The conversation was no longer about whether something was done. It was about whether it was what they wanted. The first conversation had been exhausting and adversarial. The second was useful. She said she hadn't had to argue about whether a feature worked in three weeks. It was a lesson the leader could carry to other parts of her division: teams that define quality earlier get better results later.

Testing shows where quality failed. The definition of done decides whether it fails at all.

Share With Your Team

Choose who it is for. You send it yourself from your own email; Agileon never writes to your colleagues.

Not sure where to start?

Answer five questions about where work stalls. We send back a starting plan: where to begin and what to look at first.