When you put a volt meter into an electrical circuit to measure the voltage difference between two points the internal resistance of the meter itself changes the circuit and impacts the reading. The best volt meters are designed to have very very high internal resistance, but in all circuits the resistance of the volt meter changes the circuit and impacts the measurement (circular yes!); in sensitive low voltage circuits where accuracy is most important the impact is greatest.
One of the agile development teams I’ve worked with decided that they would use the number of cards as their measure for planning, velocity and by implication productivity. Each iteration the cards from the last release were proudly pinned to the wall and counted. An interesting pattern emerged over a number of iterations: the number of bug cards completed went up, and up and up.
We spent a great deal of time trying to work out where our development, testing and release process was going wrong that we were seeing so many old bugs. Were we finding old issues with more thorough testing? Were the developers being lazy? Or was the work we were tackling just much tougher than previous projects? It turns out that the explanation was in fact very simple. Previously developers found and fixed bugs as part of their development tasks (this is quite normal), now they were writing these up as cards and running them across the board through the full prioritisation and QA process. Consciously or subconsciously the developers had been sent a signal that the number of cards was being measured, and thus they responded to and optimised their process to meet that benchmark. Issues which would have been fixed quickly while the developer was deep in their context now became tasks of their own, a much more expensive and unnecessary process.
Perhaps a better title for this post might be ‘Only measure what you truly wish for’. This is a topic which we’ll surely return to again and again, but I’ll leave you with a quote I write into the front of every one of my notebooks to remind me of a valuable truth.
“Not everything that counts can be counted, and not everything that can be counted counts.” – Albert Einstein.