Péter Érdi: RANKING. The Unwritten Rules of the Social Game We All Play

A recurring theme in our complex world pertain to the question of whether it is possible to summarize the performance of an organization faithfully with a single score? Universities, colleges and schools are complex social organizations that serve a variety of purposes, and measuring their performance is obviously delicate. What does it really mean if we say that this university is a 27th and the other one in the state the 42th ? How these numbers influence the big stakeholders of the college ranking game, students and their parents, admissions offices, and college administrators?
While University ranking became an obsession in this century, it is not a strong exaggeration to state, that everybody criticizes and simultaneously uses rankings. Ranking is and remains with us, so the best thing we can do is to understand the rules of the game. We should remember the lesson hopefully learned so far: ranking reflects the mixture of the reality and illusion of objectivity, and also subject of manipulation.

A little history

While our obsession to ranking is relatively new there are even early precursors of university. In an isolated, pioneering work published in 1863, a Czech professor of the Prague Polytechnical Institute, Carl Koristka, analyzed and compared the technical universities of the leading European countries118. The university called today Karlsruhe Institute of Technology, and known as one of the Germany leading engineering schools had the largest number (about 800) students and 50 professors. If we could believe in the numbers, the students/faculty number has been reduced to sixteen to five, since nowadays the 25; 000 students are served by 6000 academic staff. It is interesting to see that while in Karlsruhe the number of foreign students was about sixty
percent, Berlin had only two (!!) percent (seven from 374). The interval for the students/faculty ratio for the institutions for which Koristka found reliable data, was between eight and eighteen. (Koristka himself did not use the students/faculty ratio, probably it was not in the focus of attention).

James McKeen Cattell (1860-1944) was a pioneer professor in the United States, who contributed very much to the transformation of psychology from pseudoscience to legitimate science by adopting both experimental and quantitative methods. He was motivated among others by Francis Galton, who, as we remember, liked to count and measure everything. Cattel was inspired to study distinguished men of science. He requested a number of competent men in each field to rate their colleagues, or more precisely to denote the excellence with stars. Institutions, characterized by the ratio of starred scientists to the total number of faculty, were ranked. Cattel’s aim was to provide help both to potential students, and the institutions. The first edition of the American Men of Science was published in 1906, and the seventh in 1944 119. Cattel’s approach
suggested that the quality of the universities can be measured by the number of excellent faculty, and it determined our way of thinking about university ranking. The importance of ”distinguished persons” in the ranking procedure ensured the priority of the older private institutions over the newer public universities. The other early ranking systems added several more criteria. Graduate success in later life is an output measure of teaching quality, while student/faculty ratio and volumes in the library is an input measure of the resources 120.
Symbolically our modern obsession with university ranking is represented with the appearance of the US News and Report (USNWR in 1983. Mass media entered the scene. USNWR simultaneously wished to provide accessible information for students and parents, as well as increase the visibility and revenue of the magazine. Soon it turned out that the ranking is a measure of reputation, and college administrators (not necessarily admittedly) made an explicit target to rise in the U.S. News Ranking. The reputation race shifted the gear.
Now USNWR discriminates between ranks for best quality as well as for best value. USNWR calculates best value by weighting quality with 60 percent of the overall score; the percentage of receiving need-based grants accounted for 25 percent; and the average discount accounted for 15 percent. USNWR changed its methodology as a response for criticism, and combines more objective input data (resources, entering student quality) and subjective aspects of reputation. However, it is difficult to enter a race, if the rules are changing. While the US (and British) ranking systems were followed by the emergence of many national ranking systems, the race became much more exciting by the appearance of the global ranking. The three most influential global rankings are those produced by Shanghai Ranking Consultancy (the Academic Ranking of
World Universities; ARWU), Times Higher Education (THE), and Quacquarelli Symonds (QS). They measure rather research and not the teaching performance

 

Academic Ranking of World Universities 2018

Nincs magyar …

Tyranny of metrics? I was already working on this project when the closest book to my
subject was published 100. Jerry Z. Muller’s the The Tyranny of Metrics studies our obsession with metrics, and Muller lists what he calls the unintended consequences of such an obsession. Muller might be correct in his argument that a lack of social trust is the main reason that human judgments have been substituted by employing metrics for accountability and transparency. In a world where we assume the honesty, integrity, and reliability of other people, transparency could be reached by using fewer metrics.
Everybody knows stories about how metrics has been gamed. In policing, the numbers of
cases solved, crime rates, and other statistics have been manipulated to produce a better image of the performance of a police branch. In education “teaching to the tests” works against the real goal of schooling (education) in order to meet externally prescribed targets. In the health care system, we have heard anecdotes about surgeons who avoid treating risky patients so as not to reduce their performance measures. Muller also is right that there is a discrepancy between what can be measured and is worth measuring. One example is that it is easier to measure the amount of investment than it is to measure the result of an investment. While I do agree with the overwhelming majority of the examples and arguments in his book, as an ardent scientist, I can’t comply with the tone of his conclusions. Would it be a welcome development to abandon the use of metrics, rating, ranking, and any quantitative analysis? Who would then make judgments, and what would be the basis of such judgments? I think the book neglected to analyze the benefits of the accountability provided by metrics, which might overcompensate for the obvious drawbacks. Well, Professor Muller, I am ready to offer a draw.

Observers and observed The reality of science is based on objective measurements: experiments with results that are reproducible. In science, it does not happen that somebody can state and receive respect for such declarations as: “I am such a fantastic genius, I am able to make an experiment that nobody can reproduce!” In the overwhelming majority of cases in natural science, the observed phenomenon (say, the velocity of the fall of an apple from a tree) does not depend on the mental state of the observer. Even if you sleep, the apple will fall down. (And I hope the apple did not fall on your head.) In the world of the microscopic particles there is an interaction between observer and observed (but I am not writing a book on physics).
Unfortunately, humans are not apples. Observations influence human behavior. Even infants may be more prone to crying if they know it will get them what they want. Campbell’s law and Goodhart’s law are nothing more than illustrations of a famous quote attributed to the physicist Murray Gell-Mann (who received a Nobel prize for his contribution to theory of particles): “Think how hard physics would be if particles could think!” The bottom line is a triviality. Observations, measurements, and assessments reflect the past performance of people and institutions. However, people and institutions have the chance to act and react. They adopt strategies for generating a better-than-real result by manipulating information (say, if police don’t report all the crimes). However, the goal of the majority of performance assessments is to help decision makers to allocate resources. The most frequently used resource allocation strategy is to give funds to those competitors who showed a better performance, giving them a better ranking. This reactive mechanism leads the amplification of small advantages.

 

The reality and myth of measurement

The process of measurement was indispensable even in the ancient civilizations. The determination of length, mass, volume and time was crucial for supporting agriculture, construction, and trade. William Thomson (1824–1907), generally referred as Lord Kelvin, famously stated: ”When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarely, in your thoughts advanced to the stage of science.” Frederick Taylor (1856-1915) founded what is called scientific management and adopted measurement of any labor process in the production with the hope of improving productivity. This approach, called taylorism was attacked that it considers workers as cogs in the big machine of the factory, and was famously mocked in Aldous Huxley’s “Brave New World” (1932), and in Charlie Chaplin’s “Modern Times” (1936). However, it’s spirit survived. ”Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you
can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”96

The dangerous side of measurements

Donald Campbell (1916-1996) was a social scientist with an extremely broad field of interests. Campbell’s law 97 as it is commonly called, states
”The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
Charles Goodhart is an economist from the London School of Economics, and a former
member of the Bank of England’s Monetary Policy Committee stated ”that once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play that role.” Goodhart’s law 98 states that ”Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
Managers in every areas from law enforcement to health care, travel to education, have
to report numbers to characterize the performance of their organizations. There are many well-documented examples from the former Soviet Union and related countries, which could be counted as case studies to Campbell’s law. Planners set targets for the factories, emphasizing quantity rather than quality. Directors were judged on whether or not they hit their targets.
Product quality and consumer satisfaction was not a major factor. ”When five-year plans set targets in terms of tonnage, factories made things that were comically heavy—chandeliers that pulled down ceilings and roofing metal that collapsed buildings.”99

Appeal to authority We cannot say that is unreasonable to believe authorities. There is a logical model behind of appeal to authority: Assumption 1: X is an authority on a particular topic. Assumption 2: X makes a statement about that topic. Conclusion: X is probably correct. In the ideal world of scientists there is an agreement that authorities should prove their statements as rigorously as a graduate student. In politics, well . . . everybody knows this fallacy:
”An extremely credible source” has called my office and told me that @BarackObama’s birth certificate is fraud. It should be extremely credible, since he himself stated that it is . . . . How about this statement? Einstein said that E = mc2, so it is true. There is no causal relationship between who says something and whether it’s true or not. What is true that mass-energy equivalence is a general principle, and it is the consequence of some fundamental properties of time and space. (No more physics, let’s speak about ads). What is the relationship between a celebrity actor, who became famous, as Dr. Ross, and machines, which brew coffee. He says, that ”he is proud to work with the company in its commitment – that every cup its coffee has a positive impact on the world.”89

Game change in the media manipulation ”If you don’t read the newspaper, you’re uninformed. If you read the newspaper, you’re misinformed” (Mark Twain). Even in the old dayst here where threats to media objectivity. Politicians and journalists might wanted to change reality. They could exploit the fact that the media was more or less reliable. Distortion, exaggeration, fabrication and simplification was the exception, not the rule as now. Traditional authoritarians controlled of all media, and adopted censorship and ideologically oriented propaganda to maintain hegemony over their populations. In the world of the new authoritarians more sophisticated methods are employed to influence public opinion and shape political narratives. By restricting space for alternative media outlets, and ensuring the dominance of state-owned and state-friendly media assets, the new autocrats keep dissenting views out of the news and manipulate political discourse.
In the past, general interest intermediaries (think of the 1960s when there were three major news networks–ABC, NBC, and CBS–that controlled TV news) have exercised a great deal of influence over access to information, so traditional techniques of censorship have entailed shuttering newspapers, revoking broadcast licenses, or threatening (or even murdering) journalists that disagreed with a government’s agenda. But now we live in an age where ”media” has come to mean everything from CNN or NPR to one’s Facebook feed. With the decline in the relative power of general interest intermediaries and the rise in the influence of personalized media, manipulation
and censorship techniques now focus on making the entire media landscape seem
illegitimate and sowing distrust in what have historically been considered ”objective” institutions and voices.90.

The future of free speech and new form of censorship and manipulation are now hot topics in our changing world 91. A new type of manipulation seems to emerge. We read what we want to read, and it is major threat 92 If technology efficiently use filters, people get predetermined information by delivering them a personalized journal, the ”Daily Me”. So, they (we) will live in echo chambers which are means to amplify beliefs. (Remember for confirmation bias!). It would exclude to get surprising news from people living in other ”chambers”, and contribute to live in a society where people are closed by their own choice to their own mind.

 

 

Selective truth Selective facts are more dangerous than fake news. We use news to bring decisions by ranking (consciously or unconsciously our options). The media mogul Rupert Murdoch declared: ”Produce better papers. Papers that people want to read. Stop having people write articles to win Pulitzer prizes. Give people what they want to read and make it interesting.”79.
As it was discussed in the previous chapter, we are subject of confirmation bias, so we prefer to read news which fits into our mental framework. So, while traditionally news were supposed to reflect reality, now there is a news filtering mechanism which amplifies our beliefs. To put it another way, media organizations tries to find out (and data and algorithms help them to do it efficiently) what are the engaging news for us, and they feed us with such kinds of news.80.

How we react if we get selective facts from the other side? Actually I have some fresh experience. I am writing this section in July 2018 in Budapest. As you already know, I am a soccer fan, so am watching almost all games of the World Cup on the state-controlled Hungarian sport channel. In the half time there are short news, and literally all items speak over an over again about crimes immigrants committed somewhere in Europe. The Hungarian leader learned the lesson: repetition, repetition and repetition (of oversimplified and one-sided) messages!
Repetition. Lewis Caroll (real name: Charles Lutwidge Dodgson (1832-1898)) wrote in The Hunting of the Snark nonsense-poem:”What I tell you three times is true.” While English literature was probably not his strength, Hitler famously put ”no limit on what can be done by propaganda; people will believe anything, provided they are told it often enough and emphatically enough, and that contradicters are either silenced or smothered in calumny”81

We know political slogans of our time, from Yes We Can 82 to America First. In Orwell’s Animal Farm 83 Old Major repeats the same idea with slight stylistic variation to argue against the humans: ”Man is the only real enemy we have.” ”Remove Man from the scene and the root cause…is abolished.” ”Man is the only creature that consumes without producing.” ”Only get rid of Man.” More systematic psychological studies shows that repetition create the ”illusion of truth”84. My own suggestion is, please don’t repeat things without carefully checking if they are true. If you do, you are are also responsible to make a world where it is difficult to discriminate between lies and truth. So, please, please, please, think before you repeat!

Repetition. Lewis Caroll (real name: Charles Lutwidge Dodgson (1832-1898)) wrote in The Hunting of the Snark nonsense-poem:”What I tell you three times is true.” While English literature was probably not his strength, Hitler famously put ”no limit on what can be done by propaganda; people will believe anything, provided they are told it often enough and emphatically enough, and that contradicters are either silenced or smothered in calumny”81

We know political slogans of our time, from Yes We Can 82 to America First. In Orwell’s Animal Farm 83 Old Major repeats the same idea with slight stylistic variation to argue against the humans: ”Man is the only real enemy we have.” ”Remove Man from the scene and the root cause…is abolished.” ”Man is the only creature that consumes without producing.” ”Only get rid of Man.” More systematic psychological studies shows that repetition create the ”illusion of truth”84. My own suggestion is, please don’t repeat things without carefully checking if they are true. If you do, you are are also responsible to make a world where it is difficult to discriminate between lies and truth. So, please, please, please, think before you repeat!

We have seen huge scientific advances from quantum computing to space exploration during the twentieth and twenty-first centuries. It would be silly to assume that psychology did not develop during this period as well 9 . If we consider the ranking game a competition, some players are always ready to violate the rules to ensure that their priorities are well-managed. If the rules are unwritten, they are even easier to breach. In many games there are referees, umpire, judges, or arbitrators. However, “Life is a game with many rules but no referee,” as we know from Joseph Brodsky (1940-1996), the Russian-American Nobel prize-winning poet. (To provide some context, Brodsky was once asked: “You are an American citizen who is receiving the Prize for Russian-language poetry. Who are you, an American or a Russian?,” to which he answered: “I’m Jewish; a Russian poet, an English essayist – and, of course, an American citizen.”) Manipulators have the intention of gaining an advantage by adopting various tricks, from simply cheating to sophisticated propaganda techniques. Their goal is to lead the list of successful people.

5.2.1
Psychological manipulation
How to manipulate?
Appeal to fear Appealing to fear is a technique used to motivate people to take a specific
action, or support a particular policy decision, by arousing fear in listeners. As the reader knows well, this strategy has been adopted by US presidents, who have made claims like, “If we don’t bail out the big automakers, the US economy will collapse. Therefore, we need to bail out the automakers,” or “The attacks on our police, and the terrorism in our cities, threaten our very way of life.” Experts often point out that these tactics exaggerate the present threat, but the strategy persists; the current (2018) president’s “formula is very clean and uncomplicated: Be very, very afraid. And I am the cure.” 10 .
The Hungarian election campaign in 2018 had a single topic: fear. One pundit has written, It hardly seems to matter that the migration crisis has largely passed and that there are now more posters in Hungary about the danger of immigrants and refugees than actual refugees and immigrants let into the country this past year. The poster is in keeping with a campaign that has been rife with dirty tricks, false news stories, vicious personal attacks, conspiracy theories and perceived enemies all around.” 11 .

Black-or-white fallacy ”Who, you’re not with us is against us. Another US president in this
century declared: ”Either you are with us or you are with the terrorists” 12 . if you were asked which would you rather be: against the Patriot Act, or a patriot, the question implies that if you are against the Patriot Act, then you cannot be a patriot. However, if someone is not your ally does not imply that she opposes you. You can’t exclude that she might be neutral or simply undecided. If you are forced to choose among two options, and other options are eliminated, this logical fallacy.

…