Welcome to the site of Ranking

I decided to write a non-fiction book with the title and subtitle RANKING – The reality, illusion and manipulation of objectivity.  The book discusses the Hows and Whys of our love and fear of making ranks and being ranked through many real life examples to be viewed from three different angles (reality, illusion and manipulation) of objectivity. Ranking converts scientific theories to everyday’s experience by raising and answering such question as:

  • Are college ranking lists objective?
  • How to rank and rate states based on their fragility, corruption or even happiness?
  • How to find the most relevant web pages?
  • How to rank employees?

Life and society is really complex, consequently our message is not so simple such as ”Ranking is good!” or ”Ranking is bad!”. Since we permanently rank ourselves and others and are also being ranked, the message is twofold: how to prepare the possible most objective ranking and how to accept that ranking does  not necessarily reflects our real values and achievements. The reader will understand our difficulties to navigate between objective and subjective and gets help to identify and modify her place in real and virtual communities by combining our human intelligence with computational techniques.

 

Pay more in tax and be happier

While there are almost infinitely many ways to rank countries, many readers will
agree with me that one of the most important questions to answer is how happy
a country is. In 2011, the UN General Assembly initiated a project that sought to
measure the happiness of citizens of member countries. But how do we measure
the happiness of a country? The measurement is mostly based on a simple task: in
each country, a significant number of people are asked: “Please imagine a ladder,
with steps numbered from 0 at the bottom to 10 at the top. The top of the ladder
represents the best possible life for you and the bottom of the ladder represents the
worst possible life for you. On which step of the ladder would you say you personally
feel you stand at this time?”
A report issued by the UN in 2017 ranked Norway as the happiest country in the
world. (The Reader already knows that the phenomenon of “pecking order” among
chickens was discovered in Norway, and the Norewgian Magnus Carlsen has the highest
Elo number. The neurobiologists among our readers will also remember that
May-Britt Moser and Edvard Moser from Norway were awarded the Nobel Prize in
Physiology or Medicine in 2014 for discovering certain types of neurons called grid
cells, which are responsible for spatial information processing.) It was remarkable to
see the reaction of the Prime Minister Erna Solber: “even if we top this statistic now
we must continue to prioritize mental healthcare.” Actually there is no statistically
significant difference among the happiest five countries, which each received scores
around 7:5: Norway – 7:54; Denmark – 7:52; Iceland – 7:50; Switzerland -7:49 and
Finland – 7:47. The Central African Republic had the lowest score at 􀀀2:69.
In 2018, Finland took the lead, and the United States ranked 18th out of 156 countries
surveyed–—down four spots from 2017’s report. In spite of a strong economy, the US
ranks quite poorly on social measures such as life expectancy and suicide rates. Major
factors possibly contributing to this drop in ranking are the worsening of the opioid
crisis, the growing economic inequality, and the decrease in confidence in government.
Investment in mental health care is likely to correlate to average happiness. A good
proxy for investment in mental healthcare is the number of psychiatrists and psychologists working in mental health care per capita. Based on these figures, average
happiness appears to be higher in countries that invest more in mental health care.126
Like it or not, developed mental health care implies the more frequent use of antidepressants.
Increasing antidepressant utilization and decreasing national suicide rates
have been reported recently from the happiest country in the world.

Countries where we live

As we all know, humankind has organized itself into geopolitical units, called countries.
Historically, people have preferred to belong to a particular country and to share a sense of national identity. Homophily is an archaic trait: we like to spend more time with people who are like ourselves. While there are people, however, who
believe that the idea of countries, as nation-states, is outdated and the source of
conflict, countries remain a primary means of controlling people, organizing society,
and managing the distribution of wealth.
Countries are ranked and rated now by an enormous number of criteria, adopted by
hundreds of different organizations, sometimes strongly connected to specific countries
(frequently to the US). In a book about the ranking of countries, 125 authors
Cooley and Snyder identify ninety-five indices that have been introduced to evaluate
and compare states. The indices are lumped into categories, like “Business and Economics,” “Country Risk,” “Democracy and Governance,” “Environment,” “Media and
Press,” “Security Issues and Conflict,” “Social Welfare,” and “Transparency.”
A ranked list of countries based on the social welfare function defined by Amartya Sen
has been prepared annually by using data from the Central Intelligence Agency, and
another version is prepared using data from the International Monetary Found and
United Nations. (Remember: the Sen social welfare function is calculated as product
of GDP per capita and the difference between 1 and the society’s inequality measure,
and it is reported in terms of dollars per person per year.) The last published list is
from 2015:
1. Qatar | 82884
2. Luxembourg | 49242
3. Norway | 47861
4. Singapore | 43518
5. Switzerland | 42335
6. Netherlands | 34853
7. Sweden | 34443 per
8. Denmark | 33907
9. Germany | 33719
10. Iceland | 33695
11. United States | 33260
Qatar has a well-developed oil exploration industry, and the petroleum industry accounts for 60% of the country’s GDP. Its low (but rapidly increasing due to an influx
of migrant workers) population contributes to a large GDP per capita. The population
explosion due to the immigration of (young) males has produced an extreme
gender imbalance (there are only about 700; 000 women in a country of 2:5 million
people). Many immigrants, mostly involved in building the infrastructure needed for
the upcoming (well, soccer) World Cup, live in labor camps. However, since the Gini
inequality index measures income inequalities but not social inequalities, Qatar still
leads the list. Please note, that the scores of the last six countries are close to each
others, and the specific ranking does not have too much significance. (It is somewhat

Nobody likes, everybody uses: university ranking II

Demand for Ranking
Transparency, accountability and comparability There is an increased demand of transparency, accountability and comparability of the higher educational institutions from the public and the politicians 121. Ranking methodology offered a simple and easily interpretable comparison. Ellen Hazelkornin his excellent book 122 published a list of typology of transparency, accountability and comparability  instruments:
Accreditation: certification, directly by government or via an agency, of a particular HEI
with authority/recognition as an HEI and to award qualifications.
Assessment, Quality Assurance (QA) and Evaluation: assesses institutional quality processes, or quality of research and/or teaching and learning.
Benchmarking: systematic comparison of practice and performance with peer institutions.
Classification and Profiling: typology or framework of higher education institutions to
denote diversity usually according to mission and type.
College Guides and Social Networking: provides information about higher education institutions for students, employers, peers and the general public.
Rankings, Ratings and Banding: enables national and global comparison of higher education performance according to particular indicators and characteristics which set a ”norm” of achievement. The different instruments partly reflects the past performance and partly helps to plan future activity.
Heterogeneity and comprehensivity Malcolm Gladwell has already explained the nuts and
bolts of college rankings in a New Yorker article titled ”The Order of Things”. What college rankings really tell us”, published in 2011). He describes the evolution of the U.S. News ranking systems, and the difficulties of being both ”comprehensive and heterogeneous”. (Gladwell’s italics). Comprehensive means that nearly all aspects of something is included. Gladwell gives an example for heterogeneity:

”…aims to compare Penn State—a very large, public, land-grant university with a low tuition and an economically diverse student body, set in a rural valley in central Pennsylvania and famous for its football team—with Yeshiva University, a small, expensive, private Jewish university whose undergraduate program is set on two campuses in Manhattan (one in midtown, for the women, and one far uptown, for the men) and is definitely not famous for its football team.

I think from the example it is clear that to compare these two institution much more difficult than apples to oranges. We saw in Chapter 2, that it even the latter is quite difficult. As concerns comprehensivity and heterogeneity, there is a trade-off between the two characteristics.

 

Nobody likes, everybody uses: university ranking I

A recurring theme in our complex world pertain to the question of whether it is possible to summarize the performance of an organization faithfully with a single score? Universities, colleges and schools are complex social organizations that serve a variety of purposes, and measuring their performance is obviously delicate. What does it really mean if we say that this university is a 27th and the other one in the state the 42th ? How these numbers influence the big stakeholders of the college ranking game, students and their parents, admissions offices, and college administrators?
While University ranking became an obsession in this century, it is not a strong exaggeration to state, that everybody criticizes and simultaneously uses rankings. Ranking is and remains with us, so the best thing we can do is to understand the rules of the game. We should remember the lesson hopefully learned so far: ranking reflects the mixture of the reality and illusion of objectivity, and also subject of manipulation.

A little history

While our obsession to ranking is relatively new there are even early precursors of university. In an isolated, pioneering work published in 1863, a Czech professor of the Prague Polytechnical Institute, Carl Koristka, analyzed and compared the technical universities of the leading European countries118. The university called today Karlsruhe Institute of Technology, and known as one of the Germany leading engineering schools had the largest number (about 800) students and 50 professors. If we could believe in the numbers, the students/faculty number has been reduced to sixteen to five, since nowadays the 25; 000 students are served by 6000 academic staff. It is interesting to see that while in Karlsruhe the number of foreign students was about sixty
percent, Berlin had only two (!!) percent (seven from 374). The interval for the students/faculty ratio for the institutions for which Koristka found reliable data, was between eight and eighteen. (Koristka himself did not use the students/faculty ratio, probably it was not in the focus of attention).

James McKeen Cattell (1860-1944) was a pioneer professor in the United States, who contributed very much to the transformation of psychology from pseudoscience to legitimate science by adopting both experimental and quantitative methods. He was motivated among others by Francis Galton, who, as we remember, liked to count and measure everything. Cattel was inspired to study distinguished men of science. He requested a number of competent men in each field to rate their colleagues, or more precisely to denote the excellence with stars. Institutions, characterized by the ratio of starred scientists to the total number of faculty, were ranked. Cattel’s aim was to provide help both to potential students, and the institutions. The first edition of the American Men of Science was published in 1906, and the seventh in 1944 119. Cattel’s approach
suggested that the quality of the universities can be measured by the number of excellent faculty, and it determined our way of thinking about university ranking. The importance of ”distinguished persons” in the ranking procedure ensured the priority of the older private institutions over the newer public universities. The other early ranking systems added several more criteria. Graduate success in later life is an output measure of teaching quality, while student/faculty ratio and volumes in the library is an input measure of the resources 120.
Symbolically our modern obsession with university ranking is represented with the appearance of the US News and Report (USNWR in 1983. Mass media entered the scene. USNWR simultaneously wished to provide accessible information for students and parents, as well as increase the visibility and revenue of the magazine. Soon it turned out that the ranking is a measure of reputation, and college administrators (not necessarily admittedly) made an explicit target to rise in the U.S. News Ranking. The reputation race shifted the gear.
Now USNWR discriminates between ranks for best quality as well as for best value. USNWR calculates best value by weighting quality with 60 percent of the overall score; the percentage of receiving need-based grants accounted for 25 percent; and the average discount accounted for 15 percent. USNWR changed its methodology as a response for criticism, and combines more objective input data (resources, entering student quality) and subjective aspects of reputation. However, it is difficult to enter a race, if the rules are changing. While the US (and British) ranking systems were followed by the emergence of many national ranking systems, the race became much more exciting by the appearance of the global ranking. The three most influential global rankings are those produced by Shanghai Ranking Consultancy (the Academic Ranking of
World Universities; ARWU), Times Higher Education (THE), and Quacquarelli Symonds (QS). They measure rather research and not the teaching performance

 

The importance and the difficulties of measuring society II

Tyranny of metrics? I was already working on this project when the closest book to my
subject was published 100. Jerry Z. Muller’s the The Tyranny of Metrics studies our obsession with metrics, and Muller lists what he calls the unintended consequences of such an obsession. Muller might be correct in his argument that a lack of social trust is the main reason that human judgments have been substituted by employing metrics for accountability and transparency. In a world where we assume the honesty, integrity, and reliability of other people, transparency could be reached by using fewer metrics.
Everybody knows stories about how metrics has been gamed. In policing, the numbers of
cases solved, crime rates, and other statistics have been manipulated to produce a better image of the performance of a police branch. In education “teaching to the tests” works against the real goal of schooling (education) in order to meet externally prescribed targets. In the health care system, we have heard anecdotes about surgeons who avoid treating risky patients so as not to reduce their performance measures. Muller also is right that there is a discrepancy between what can be measured and is worth measuring. One example is that it is easier to measure the amount of investment than it is to measure the result of an investment. While I do agree with the overwhelming majority of the examples and arguments in his book, as an ardent scientist, I can’t comply with the tone of his conclusions. Would it be a welcome development to abandon the use of metrics, rating, ranking, and any quantitative analysis? Who would then make judgments, and what would be the basis of such judgments? I think the book neglected to analyze the benefits of the accountability provided by metrics, which might overcompensate for the obvious drawbacks. Well, Professor Muller, I am ready to offer a draw.

Observers and observed The reality of science is based on objective measurements: experiments with results that are reproducible. In science, it does not happen that somebody can state and receive respect for such declarations as: “I am such a fantastic genius, I am able to make an experiment that nobody can reproduce!” In the overwhelming majority of cases in natural science, the observed phenomenon (say, the velocity of the fall of an apple from a tree) does not depend on the mental state of the observer. Even if you sleep, the apple will fall down. (And I hope the apple did not fall on your head.) In the world of the microscopic particles there is an interaction between observer and observed (but I am not writing a book on physics).
Unfortunately, humans are not apples. Observations influence human behavior. Even infants may be more prone to crying if they know it will get them what they want. Campbell’s law and Goodhart’s law are nothing more than illustrations of a famous quote attributed to the physicist Murray Gell-Mann (who received a Nobel prize for his contribution to theory of particles): “Think how hard physics would be if particles could think!” The bottom line is a triviality. Observations, measurements, and assessments reflect the past performance of people and institutions. However, people and institutions have the chance to act and react. They adopt strategies for generating a better-than-real result by manipulating information (say, if police don’t report all the crimes). However, the goal of the majority of performance assessments is to help decision makers to allocate resources. The most frequently used resource allocation strategy is to give funds to those competitors who showed a better performance, giving them a better ranking. This reactive mechanism leads the amplification of small advantages.

 

The importance and the difficulties of measuring society I.

The reality and myth of measurement

The process of measurement was indispensable even in the ancient civilizations. The determination of length, mass, volume and time was crucial for supporting agriculture, construction, and trade. William Thomson (1824–1907), generally referred as Lord Kelvin, famously stated: ”When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarely, in your thoughts advanced to the stage of science.” Frederick Taylor (1856-1915) founded what is called scientific management and adopted measurement of any labor process in the production with the hope of improving productivity. This approach, called taylorism was attacked that it considers workers as cogs in the big machine of the factory, and was famously mocked in Aldous Huxley’s “Brave New World” (1932), and in Charlie Chaplin’s “Modern Times” (1936). However, it’s spirit survived. ”Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you
can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”96

The dangerous side of measurements

Donald Campbell (1916-1996) was a social scientist with an extremely broad field of interests. Campbell’s law 97 as it is commonly called, states
”The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
Charles Goodhart is an economist from the London School of Economics, and a former
member of the Bank of England’s Monetary Policy Committee stated ”that once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play that role.” Goodhart’s law 98 states that ”Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
Managers in every areas from law enforcement to health care, travel to education, have
to report numbers to characterize the performance of their organizations. There are many well-documented examples from the former Soviet Union and related countries, which could be counted as case studies to Campbell’s law. Planners set targets for the factories, emphasizing quantity rather than quality. Directors were judged on whether or not they hit their targets.
Product quality and consumer satisfaction was not a major factor. ”When five-year plans set targets in terms of tonnage, factories made things that were comically heavy—chandeliers that pulled down ceilings and roofing metal that collapsed buildings.”99