The Tyranny of Metrics, by Jerry Z. Muller
THE author cites as primary reason for many contemporary organisations functioning less well than they ought to, with diminishing productivity and frustrated employees, as the “metric fixation”.
Professor Jerry Z Muller teaches at the Catholic University of America in Washington. His interest in the effects of metrics was sparked by his institution (as are other academic institutions) being rewarded or penalised by government funders, based on how they performed against a set of imposed metrics.
Underlying this approach is the dictum of the 19th century physicist Lord Kelvin: “If you cannot measure it, you cannot improve it.”
More recently, we have the Tom Peters’ motto, “What gets measured gets done.” From this, many have concluded that only that which can be counted, really counts.
This book is not about the evils of measuring, but about excessive and inappropriate measurement. Muller’s critique ranges from medicine to education at all levels and ranges from policing and public services to charitable organisations and business.
I will focus only on business.
The fixation on metrics is the aspiration to replace judgement based on experience with standardised measurement. Judgement is demeaned as being “personal, subjective, and self-interested”.
Clearly, decisions based on big data are useful and necessary. This is obvious when the experience of any single practitioner, is likely to be too limited to develop an intuitive and reliable feel for the most effective method or remedy.
Statistical analysis can sometimes discover that neglected characteristics are more significant than the intuitive understanding of experience believed them to be.
Checklists are hugely valuable in fields as different as airlines and medicine. However, these checklists are standardised procedures for action under specific conditions. I am a strong proponent of checklists.
So, what is the problem? It is “the belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data”.
To make matters worse, the ‘metric fixation’ believes that the best way to motivate people in organisations, is by attaching monetary or reputational rewards and penalties for their performance.
Unintended negative consequences
The ‘metric fixation’ has unintended negative consequences. “Not everything that can be counted counts, and not everything that counts can be counted.”
Most organisations have multiple purposes. What is measured and rewarded tends to become the focus of attention, at the expense of other essential goals. For starters – gathering metrics seem to waste more and more time and resources.
Muller lists seven recurring flaws that will assist in separating the necessary from the misleading or distracting. The list serves as an alert.
The seduction to measure the most easily measurable is ever present. However, what is most easily and simply measured is rarely what is most important.
The important and desirable outcomes are most often complex. Focusing measurement on just one responsibility or goal leads to deceptive results.
Organisations are inclined to measure what they’ve spent, rather than what they have produced - the process rather than product. This is especially true where the product is complex and multi-faceted.
When information is standardised, it can seem organised and simplified. This makes it easy to compare people and institutions. The downside is that this sort of measurement often hides the degraded quality of the information, now that it is stripped of context, history, and meaning.
‘Gaming’ the metrics can be achieved in many ways. The successes can be creamed off by excluding cases where success is more difficult to achieve. A sales closing rate can be 100% only because the salesperson is not prospecting widely.
Numbers can be improved by lowering standards. Pass rates are higher when the standard for passing is lowered – even though the purpose of rating passes is to educate.
Omission or distortion of data
The results can also be improved through omission or distortion of data. Police forces can “reduce” crime rates by booking felonies as misdemeanours, or by deciding not to note reported crimes at all.
‘Cheating’ is one step beyond gaming the metrics. It is a phenomenon whose frequency tends to increase, based on to the importance of the metric being reported, to the individual or business.
The metric fixation has had a long history in business. It started in 1911 when an American engineer, Frederick Taylor, coined the term “scientific management”.
His context was pig iron in factories, and his goal was to increase efficiency by standardising and speeding up work on the factory floor to create mass production.
We have come a long way since 1911. Many of our institutions are large; they are almost all complex, made up of dissimilar parts, that defy a Taylorist perspective.
In so many parts of business, we make decisions despite having limited time and ability to deal with the information overload.
Metrics are a tempting means to deal with matters beyond our immediate comprehension. However, this encourages a distrust of the experience and judgement of employees. Not to mention that all the time spent reporting, meeting, and coordinating, leaves little time for actual doing.
The autonomy of those lower in the organisational hierarchy who may have doubts about metric-based innovations are too often dismissed as irrational, or as having a self-interested “resistance to change”.
Intangible factors aren’t easily quantified if the work is not repetitive, uncreative, and involves standardised commodities or services. In our lifetime, technologies such as robotics and artificial intelligence will make such jobs rare.
It can be dangerously misleading to use metrics to measure the ability to come up with new ideas and better ways to do things.
Similarly, it can be dangerously misleading to use metrics for the exchange of ideas and resources with colleagues; for the ability to engage in teamwork, to mentor subordinates, to relate to suppliers or customers, and more.
Much to ponder.
Readability: Light ----+ Serious
Insights: High -+--- Low
Practical: High ----+ Low