James Newkirk and Brad Wilson have co-authored a post where they present an novel (to me) idea about code coverage metrics, e.g. “my code is 80% covered by my unit tests according to NCover or VSTS”.
The traditional way of viewing code coverage is as a scalar value. It’s simply expressed as a percentage answering what portion of your production code is touched by your test code. This is the mental model I’m used to. Under this paradigm I’ve always viewed code coverage percentage as a somewhat empty metric; more of a rule of thumb than a if-you-can’t-measure-it-you-can’t-manage-it kind of thing.
Just because you have 100% coverage does not mean that your tests are any good. Test code, like all code, needs to be examined, reviewed, and punished when it’s bad. Yet another good argument for collective code ownership.
I’m not sure anyone would argue that code coverage is as “hard”, reliable, or objective a metric as Iteration Velocity, Feature-Defect Ratio, Estimation Accuracy, etc. Code Coverage simply tells you “tests are being written”.
James and Brad present the metric as a vector. Really, go read the post, it’s a cool idea and I won’t spoil the killer metaphor they give here. Ultimately they’re saying what’s being ignored is the change in coverage percentage (Code Coverage Delta?) across integration builds or whichever event you choose to collect and distribute the metric.
Interesting.
Still I think the metric is overhyped, though the vector approach might increase its value by properly recasting the number as a heuristic. Now we have a trigger to 1) have a conversation, or 2) take a closer look, or 3) loosely gauge if team members are following through with good testing practices.
2 Comments
Code Coverage is a useful thermometer. It is not the actual ‘health’ per-say - but a decent aggregate indicator of health. In a nutshell - if you have 0% coverage - you are dead. If you have coverage of something - your are better than dead. If you - in general have coverage in a good area (say 70%) then you are probably doing something alright.
Yeah, I’m with you.
I’m not saying one should run without or ignore the metric and agree whole heartedly that it is a “decent indicator”, but I do think that the number tells you almost absolutely nothing of the quality of the coverage.
Let’s say a lot of your coverage comes from stuffing/fetching values from simple, pass-through mutators/accessors. What do those points rolling up into the percentage tell you then? Even with 100% coverage (through proper exclusion) there’s no guarantee that the tests are any good or represent a reasonable set of assertions.
All I’m saying is: getting to the top of the code quality hierarchy of needs comes down to difficult-to-measure things and human factors such as testing skill, pair programming, code review, etc. Once the people get there I’m curious to see if the metric expressed as a delta (iteration over iteration or day over day) might not be another good thing for our bag of tricks.
One Trackback/Pingback
[...] got pretty good coverage (despite my mixed feelings on the code coverage metric) on the library but, still, there’s this one class with nothing but HttpContext this and [...]
Post a Comment