Test Coverage Issues

What do we mean when we talk about “test coverage”?

I see two main areas:

  1. code coverage (frequently by automated unit level tests)
  2. how much of the application has been tested end-to-end (frequently manually)

Code Coverage

This is a topic with a lot of pre-existing discussion and tooling. For me, as with a lot of things in software, what good looks like “depends”. However, one key thing to not is that percentage of lines of code covered by unit tests doesn’t equate to quality of those tests, or indeed the code under test.

It can be a useful metric, but only when used in conjunction with others.

One of the issues is that here is that of combining coverage across languages and ensuring that you’ve not missed anything but also do you provide one number or multiple per-language?

It’s important to know what’s being reported, is it: statement, decision or branch coverage? (It’s likely just statement).

The DO-178B guideline is worth a look.

Test Coverage

The less well defined, and probably less useful, metric is “test coverage”.

I’ve seen this defined as:

 the number of test cases executed
-----------------------------------
  the total number of test cases

Unfortunately this is largely useless, apart from a measure of progress through running tests in a given reporting cycle.

It doesn’t give any idea into the (presumed) ultimate goal which is confidence in our testing i.e. how confident are we that releasing this code will “work”.

To me it’s just a measure of how many tests have you run out of how many test cases did you think of which on its own has very little to do with quality etc. Also, as software grows this denominator will grow and even if you’ve automated them all is it worth running the tests each cycle?

Maybe.

Conclusion

Overall, I’d consider using other metrics, or at least being clear on what you hope the above metrics are showing and then combining them with others; such as the DevOps metrics.

tags testing