How many tests are enough for this piece of code?

Let’s look at the following Ruby code, but it could be in any language, as the logic is more important than the actual code.

I think of looking at the code and focusing on the line coverage, we can say that two tests might be enough.

But this also might be misleading. We can see that if we want to test the entire truth table, then we need 4 test cases:

There are various coverage criteria that we can apply and each one will result in a different number of test cases:

How do we choose between these coverage criteria?

We choose based on risk and project context.

Choosing criteria taking risk into consideration

When talking about the risk, let’s use a simple graph having a Y axis going from low risk to high risk.

Branch coverage

In this case, for low risk, we have branch coverage. That is just making sure the entire decision (the entire expression used in the if = (A ||B)) is one time true and one time false. We could execute, for example, A = true, B = does not matter and A = false, B = false, and that will achieve 100% branch coverage.

Condition Coverage

Then, as the risk increases, we can apply condition coverage. That means ensuring that each independent condition (A or B) is both accurate and false. In this case, we need 2 test cases:

• A = false, B = true
• A = true, B = false

You can notice that actually you might want to make also the decision (the entire IF) true and false so you might choose 3:

• A = true, B = does not matter
• A = false, B = true
• A = false, B = false

This is because of the short circuit logic that most programming languages have where if for example in case of OR if the first condition is true, the other ones are not evaluated.

Modified Condition Decision Coverage

The more the risk, the more we want to make sure to cover. In this case, we want to choose so that we can also check how each condition affects the decision.

So we might choose the following test cases:

• A = false, B = false => A is decisive and B is decisive
• A = true, B = false => A is decisive, B is not
• A = false, B = true => A is not decisive, B is decisive

Truth table

Of course if we want to cover all combinations, we have to make the entire truth table:

• A = true, B = false
• A = true, B = true
• A = false, B = false
• A = false, B = true

And this one will be choosen when the risk is high as we want to cover the possible logical cases here.

Choosing criteria, taking features or the project in consideration

Imagine that you are implementing a feature that is related to compliance with a standard or legal compliance.

In this case, you should apply either a truth table or modified condition decision coverage as they have higher chances of catching more bugs and exercises more paths in your code.

But if you are working on something that is medium or low from the point of view of the end-users or clients, you might want to apply condition coverage or even just branch coverage.

Good Enough Testing means matching coverage to risk

A core concept in Good Enough Testing is adapting to the risk and project context and choosing the test cases so that it lowers the risk and it covers what’s important in our project for the end-users, for the company and for the team.

That means being systematic so we can rely on the test suite, focusing on what’s important to avoid overtesting or under-testing, and keeping it lean so the test suite is easy to maintain.

Don’t overtest. Don’t guess. Use structure.