What is good enough testing?

When I was trying to search for this website online and see how Google renders it, I discovered that James Bach coined the term “Good Enough Testing” in 1998.

I know about James Bach from teaching Exploratory testing techniques when I delivered the Certified ISTQB Foundation Level course so I think it is important to review the paper he published and see if there is something that I can use in my workshop.

James’ paper available online at James’ website sets an excellent base for thinking about Good Enough Testing.

What is Good Enough Testing?

He shares there the following definition for Good Enough Testing:

Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product.

James Bach. 1998

I agree strongly with the definition; it feels more explicit than mine.

The definition I use in the workshop is the following:

Considering the level of risk, we define coverage criteria that will drive the test design so that we can achieve 100% coverage with the minimum number of test cases.

My workshops are focused on developers, and I try to show a series of tools or test design techniques that they can pick and use depending on the feature, the risk, and the time. I talked about the time axis in the workshop, but I see it closely connected with the risk.

The main idea is that we can use systematic test design techniques, and based on the level of risk and/or time constraints, we can apply the techniques, strict or loose. We can choose the maximum coverage or pick a coverage criteria that fits the current context.

The second point is that my workshop is dedicated to developers starting from the assumption that the purpose of testing in their specific case is:

• To verify that the code works as expected

• To document the code behavior

There are also other important objectives for testing (like finding bugs and validating that the solution is helping the end-user), but I think those first 2 are closer to developers and a good fit to get started.

I plan to create a very concrete and practical workshop, so I will continue to use my own due to the nature of the workshop. There is not enough time to discuss how each part of the context can drive the testing and influence objectives, techniques, and execution. But I will definitely present and talk a bit also about James’ one.

Basic questions

In my workshop, I start with some questions that will drive what we learn:

1. How do you know how many tests are enough?
2. How do you know if your tests cover enough business logic?
3. How confident are you that your tests cover the critical parts of your code?

And the main idea is that we want to have a systematic approach to answering them.

A systematic approach means finding a repeatable process or heuristic that does not depend on the developer’s experience. We want to predict the total number of test cases and ensure we can always identify and explain the current criteria.

In the same paper “A Framework for Good Enough Testing”, James starts with the following fundamental question:

How do I know if I’m doing, or have done, enough of the right type of testing?

James Bach. 1998

He adds more questions that will help assess the value of the testing:

How accurate and complete is it? How reasonable is it? Is it within project constraints? Is there a good return on investment, such as the extent of information gained per test? How well does the assessment serve the project and the business? Is it soon enough to be useful?

James Bach. 1998

These are very useful questions to consider and would help create a testing strategy. James' chapter "Components of Assessment" covers various aspects of the product, testing costs, decision-making, and timing and in case you are preparing a testing strategy or a document about how to adapt testing to various project realities, I invite you to read it and try to answer all those questions.

How to answer all these questions about testing

Unfortunately, there is no objective or rigorous calculus for answering this question, but we can identify what to consider in attempting to answer it. We can begin to build at least a heuristic framework around the problem.

James Bach. 1998

I agree that it is hard to create a generally applicable or objective way to answer these questions for all projects and contexts and this is why I narrowed down my focus in the workshop to help developers have a couple of test design techniques and teach them how to use and adapt them when trying to test their own code.

While not enough to ensure good quality, it could be - in case of an engineering or development department - the base or starting point of good-quality product engineering. It provides the foundation for developing stronger and more user-focused quality processes for such department.

References

[James Back, 1998] A Framework for Good Enough Testing. Computer 31, 10 (October 1998), 124–126. https://doi.org/10.1109/2.722304 - Full PDF available at satisfice.com