Tyranny of metrics? I was already working on this project when the closest book to my
subject was published 100. Jerry Z. Muller’s the The Tyranny of Metrics studies our obsession with metrics, and Muller lists what he calls the unintended consequences of such an obsession. Muller might be correct in his argument that a lack of social trust is the main reason that human judgments have been substituted by employing metrics for accountability and transparency. In a world where we assume the honesty, integrity, and reliability of other people, transparency could be reached by using fewer metrics.
Everybody knows stories about how metrics has been gamed. In policing, the numbers of
cases solved, crime rates, and other statistics have been manipulated to produce a better image of the performance of a police branch. In education “teaching to the tests” works against the real goal of schooling (education) in order to meet externally prescribed targets. In the health care system, we have heard anecdotes about surgeons who avoid treating risky patients so as not to reduce their performance measures. Muller also is right that there is a discrepancy between what can be measured and is worth measuring. One example is that it is easier to measure the amount of investment than it is to measure the result of an investment. While I do agree with the overwhelming majority of the examples and arguments in his book, as an ardent scientist, I can’t comply with the tone of his conclusions. Would it be a welcome development to abandon the use of metrics, rating, ranking, and any quantitative analysis? Who would then make judgments, and what would be the basis of such judgments? I think the book neglected to analyze the benefits of the accountability provided by metrics, which might overcompensate for the obvious drawbacks. Well, Professor Muller, I am ready to offer a draw.
Observers and observed The reality of science is based on objective measurements: experiments with results that are reproducible. In science, it does not happen that somebody can state and receive respect for such declarations as: “I am such a fantastic genius, I am able to make an experiment that nobody can reproduce!” In the overwhelming majority of cases in natural science, the observed phenomenon (say, the velocity of the fall of an apple from a tree) does not depend on the mental state of the observer. Even if you sleep, the apple will fall down. (And I hope the apple did not fall on your head.) In the world of the microscopic particles there is an interaction between observer and observed (but I am not writing a book on physics).
Unfortunately, humans are not apples. Observations influence human behavior. Even infants may be more prone to crying if they know it will get them what they want. Campbell’s law and Goodhart’s law are nothing more than illustrations of a famous quote attributed to the physicist Murray Gell-Mann (who received a Nobel prize for his contribution to theory of particles): “Think how hard physics would be if particles could think!” The bottom line is a triviality. Observations, measurements, and assessments reflect the past performance of people and institutions. However, people and institutions have the chance to act and react. They adopt strategies for generating a better-than-real result by manipulating information (say, if police don’t report all the crimes). However, the goal of the majority of performance assessments is to help decision makers to allocate resources. The most frequently used resource allocation strategy is to give funds to those competitors who showed a better performance, giving them a better ranking. This reactive mechanism leads the amplification of small advantages.
From RANKING to REPAIR! 