On Software Development Metrics

As software developers, we need to be careful with metrics. I think there is an understanding that it's possible to cause more harm than help with an ill-chosen approach to metrics. One of the concerns is metrics that are susceptible to gaming. To me, a concern at least as great as gaming is measuring the wrong things.

The primary opportunity for measuring the wrong thing is by measuring mechanisms instead of results. For example, measuring pairing is a measuring a mechanism. Measuring the degree of siloing is measuring a result. Measuring testing is measuring a mechanism. Measuring code quality, but better yet product quality is measuring a result. It's the results that we care about more than the mechanisms. The mechanisms are a means to an end, not the end in themselves.

It's critical to measure the result rather than the mechanism. The first reason for this is that it's less susceptible to gaming. Consider measuring the number of tests versus the number of support calls received. Certainly, both can be gamed. But it's far easier to artificially jack up the number of tests. The real desire is to produce a system of great quality, which is subjective. It's harder to measure these subjective things, but it's worth it.

The second reason is that if we measure mechanisms, we'll miss important components of producing a quality system. So, for example, tests are a mechanism that help us deliver quality systems, but not the only mechanism. What we really care about is the quality of the delivered product. What happens if we measure the desired results instead of the means to achieve that result? First, it's harder to measure, and the outcome is more subjective. But, by measuring that, we also indirectly measure all those little things that we do as developers to make sure we don't get those 2AM calls, such as perusing the code a bit before check-in, or being well-read on pitfalls and benefits of various patterns.

The third reason for measuring the result instead of the mechanism is that measuring mechanism creates a box to think in. To make a trivial example, if we take as a metric the number of JUnit tests, we'll never be free to consider alternatives. We'll always create JUnit tests, because that's what measured. When the next great thing comes along, we'll be slower to adopt it, since it's not what we're measuring. We're thinking in a box. If we're measuring results, we will be more inclined to adopt new techniques as they come along to the extent that they seem to provide a real contribution to product quality.

It's easier to measure mechanisms than results. The main reason for this is that mechanisms tend to be more quantifiable than subjective results. The ease of measuring mechanisms is why most companies do it this way, and remain mediocre. The rule of thumb is this: You'll get more of what you measure. If you want more of a certain technique, measure it, and you'll get more of it. If you want more product quality, measure that instead - whatever it takes - and you'll get more of that. When it comes down to brass tacks, you don't want more of certain mechanisms, you want better results.