Code Coverage as a Metric

James Newkirk and Brad Wilson have co-authored a post where they present an novel (to me) idea about code coverage metrics, e.g. “my code is 80% covered by my unit tests according to NCover or VSTS”.

The traditional way of viewing code coverage is as a scalar value. It’s simply expressed as a percentage answering what portion of your production code is touched by your test code. This is the mental model I’m used to. Under this paradigm I’ve always viewed code coverage percentage as a somewhat empty metric; more of a rule of thumb than a if-you-can’t-measure-it-you-can’t-manage-it kind of thing.

Just because you have 100% coverage does not mean that your tests are any good. Test code, like all code, needs to be examined, reviewed, and punished when it’s bad. Yet another good argument for collective code ownership.

I’m not sure anyone would argue that code coverage is as “hard”, reliable, or objective a metric as Iteration Velocity, Feature-Defect Ratio, Estimation Accuracy, etc. Code Coverage simply tells you “tests are being written”.

James and Brad present the metric as a vector. Really, go read the post, it’s a cool idea and I won’t spoil the killer metaphor they give here. Ultimately they’re saying what’s being ignored is the change in coverage percentage (Code Coverage Delta?) across integration builds or whichever event you choose to collect and distribute the metric.

Interesting.

Still I think the metric is overhyped, though the vector approach might increase its value by properly recasting the number as a heuristic. Now we have a trigger to 1) have a conversation, or 2) take a closer look, or 3) loosely gauge if team members are following through with good testing practices.