Welcome to the site of Ranking

I decided to write a non-fiction book with the title and subtitle RANKING – The reality, illusion and manipulation of objectivity.  The book discusses the Hows and Whys of our love and fear of making ranks and being ranked through many real life examples to be viewed from three different angles (reality, illusion and manipulation) of objectivity. Ranking converts scientific theories to everyday’s experience by raising and answering such question as:

  • Are college ranking lists objective?
  • How to rank and rate states based on their fragility, corruption or even happiness?
  • How to find the most relevant web pages?
  • How to rank employees?

Life and society is really complex, consequently our message is not so simple such as ”Ranking is good!” or ”Ranking is bad!”. Since we permanently rank ourselves and others and are also being ranked, the message is twofold: how to prepare the possible most objective ranking and how to accept that ranking does  not necessarily reflects our real values and achievements. The reader will understand our difficulties to navigate between objective and subjective and gets help to identify and modify her place in real and virtual communities by combining our human intelligence with computational techniques.

 

Shy versus succesful

As I am thinking on marketing I read a blog Shy And Successful: 11 Reasons Why The Introvert Has The Edge 

Should I upload my close peer Brian Castellani’s mail? While I understand that shy people LESS LIKELY TO MAKE FOOLS OF THEMSELVES SOCIALLY, I take the risk to seem (and to be) fool. Here is the  (somewhat edited) mail:

1. Brilliant!  brilliant!  brilliant!  I am not kidding you.
2. I really enjoyed how you mixed stories of growing up in Budapest and other personal stuff in order to illustrate the key points you were trying to make.
3. Which leads to my main point: this is actually an incredible sociological study of the big data socio-cybernetic globalised world inwhich we presently live, and how the virtual world of social media has become so infused into our existence, that it has even overtaken science and academia.  I could not stop reading.  ranking, ranking, and more ranking.  i did not realise how essential it is to globalised life today!
4. The other thing you did very well was overturn the arrogance and over confidence of so many of our complexity colleagues who pretend like the new science cannot be fooled by its own numbers and therefore can show us the pathway to success.

[Brian, I deleted this paragraph: you criticized an immediate bestseller, and probably you are right, but I did not ask for your permission to make it open…]
5. All of which takes me to the next point:cognitive bias.  Your chapters dealing with cognitive bias and group think and bounded rationality and your usage of philosophy and cognitive science to demonstrate the underlying psychology of ranking was also excellent!
6. I also liked how you showed readers how science actually works, and the immense difficulty surrounding the politics of academic knowledge.
7. Also, it was very clear you put a lot of time and care into the
writing.  The writing is very good and also, while it was personal, it was not glib or contrived.  You kept to the ideas, which was great.
8. I have no real criticisms.  The only thing I noticed here and there were some typos, but that will not be an issue once the editors clean up things.

 

Foreword by Scott E Page

In this brilliant, wide ranging book, Péter Érdi, an award winning teacher and scholar, takes up the phenomena of rankings and ratings (I’ll get to the difference in a moment). A computational scientist, Érdi proves equally skilled as a social observer revealing deep implications of the ubiquitous rankings and ratings created by social and mainstream media. We assign far to much credibly to numerical rankings that may, at their core, be subjective impressions. Even more troubling, as we change our behavior to move up those lists, we allow ourselves to be manipulated by the rankings.

Rankings and these behavioral responses to them occur across the socio-technical landscape. To grasp the breadth of his inquiry, take a moment and leaf through the index. Here’s one sampling taken back to front: the Wong Baker pain scale, US News and World Report University Rankings, Scar (yes, from the {\em Lion King}), recommendation letters, the illusion of objectivity, the Hungarian National Soccer Team, Erdős numbers, ELO Chess Ratings, and Campbell’s Law. The A’s alone include Jane Austen, Aristotle, and Arrow’s Impossibility Theorem.

That sampling of topics only hints at the fun in store for you. Throughout, Péter ‘s fertile, lively mind is on full display. You are in for a treat. Though the book takes up technical topics, Péter’s writing manages to be bright, funny, and clear. By book’s end, many readers may contemplate catching the train to Kalamazoo or Budapest with the hopes of meeting up with Péter to learn more about preferential attachment mechanisms, bounded rationality, social neuroscience, the psychology of list making, or the applications of network statistics. For those of you who know Péter, particularly his former students, reading the book will remind you of his boundless, generous curiosity. The book like Péter, is informative, deep, thought provoking, and joyful.
The best rankings rely on objective criteria. Rankings of the tallest buildings, largest Northern Pike, and fastest motorcycles can be accepted on face value. However, even objective criteria may, when viewed under a microscope, contain elements of subjectivity. The official height of a building includes the building’s towers if they are integral to the building. The spire on the Freedom Tower in New York counts while the two antennae atop the Willis Tower in Chicago do not. The integral portion of a building lies in the eye of the beholder. And that’s where the problems start, with the inclusion of subjectivity.

Subjectivity allows us to rank as we see fit. In the movie version of Thomas Wolfe’s “The Right Stuff”, a reporter asks the astronaut Gordon Cooper, played by a young Dennis Quaid, to name the best pilot he ever saw. Cooper first rambles about pictures on a wall in a place that no longer exists and hurtling steel, all as a prelude to naming the Chuck Yeager. When Cooper realizes that the press wants a story and has little interest in the truth of the matter, he breaks into a toothy smile and says, “ who’s the best pilot, I ever saw? well, uh, your lookin’ at him.”

{\em Air and Space Magazine} would beg to differ. They do not rank Gordon Cooper in the top ten, though Yeager does rank. \’Erdi would be quick to point out that both Cooper’s ranking and the magazine’s, just like the ubiquitous rankings we find on the World Wide Web—the top ten beaches, the top eight Belgian ales, and the top seven dog breeds—-are subjective. Some person, or group of people, made up an ordering and then justified it using criteria derived after the fact. Nevertheless, these rankings confer a degree of authority—ah the power of numbers.

Yet, as Érdi shows, in most of the important cases, objective rankings are not possible. Permit me a brief foray into formalism. Formally, a ranking is a {\em complete, asymmetric, and transitive relation}. Complete means that it compares any two things. Asymmetric means that it ranks each item either above or below every other: either you like beets more than carrots or carrots more than beets. Transitive means that if A is preferred to B, and B to C, then A must also be preferred to C.
As logical as transitivity may seem, it can be violated by collections of rankings. In a Condorcet Triple, a majority rule vote of three people each of whom has transitive preferences, results in A defeating B, B defeating C, and C defeating A. Majority rule voting becomes an instantiation of the Rock–Paper–Scissors game. In other words, even if each person has a consistent ranking, that in no way implies that a collective ranking exists.

A similar problem arises if the items we wish to rank possess multiple dimensions. Magazines rank restaurants by evaluating the quality of their food, their ambiance, and the professionalism of their staff. They then assign numbers to each restaurant on each dimension and sum them to produce a rating. Out of a total score of 30, one restaurant may score 28, while another scores 27. These numbers, as \’Erdi points out, are subjective. One person’s five out of five may be another person’s four out of five. What appears scientific is largely made up.

In fairness, often the scores are a mix of objective and subjective. Such is the case in the US News and World Report’s college rankings which take into account the number of classes a college offers with fewer than nineteen students and the overall faculty to student ratio (both objective) as well as ranking by deans (subjective). To create ratings (which can then be simplified to a ranking), US News then attaches a weight to each of these criteria. How do they come up with the weights? Again, those are just made up based on common sense. So once again, what appears scientific is in fact subjective.

An immediate consequence of this method is that a college can improve its ranking by limiting enrollments in some classes to only nineteen students. Doing so improves the school’s ranking, Keep in mind that no empirical evidence supports a significant loss in learning from adding a twentieth student. US News just choose the number nineteen. To see the pernicious effects of just this one ranking criterion go to almost any college web page and you will see that they advertise the number of classes with nineteen or fewer students. Colleges prevent students from taking classes (sorry, you’re number twenty) so as not to hurt their rankings.

It follows, paradoxically, that even well intentioned attempts to identify the best of us may bring out the worst in some of us, as we distort ourselves to improve our rankings. Thus, the greater importance we attach to these mostly subjective rankings, the more we produce behavioral distortions. With this book, Professor \’Erdi has done us a service. He has taught us to think more deeply, yet done so with a captivating examples and a light hand.
Scott E Page
Leonid Hurwicz Collegiate Professor of Complex Systems, Political Science, and Economics, University of Michigan-Ann Arbor. External Faculty, The Santa Fe Institute

Fake reviews: it happens, but maybe can be filtered

The Bellgrove case

In the travel industry, TripAdvisor is a leader. It’s the yardstick for reviews and comparison for hotels and excursions. Everybody knows anecdotal evidence about the major problems with the service. Every systems can be gamed, and there are wide discussion how to cope with the huge problem induced by fake reviews.
One of the famous Tripadvisor incident is related to Bellgrove Hotel in Glasgow, Scotland, it was more the product of jokes than the intention of generating fake reviews.  The hostel served about 150, mostly homeless, unemployed men, some of them had drug and alcohol problems, so it did not have a wonderful reputation. In 2013 a number of jokers gave it a five-star rating, and a certain point Bellgrove made the top 100 of TripAdvisor’s best places to stay! I think TripAdvisor reacted properly: ”As this property is a homeless shelter, and therefore doesn’t meet our listing guidelines, the listing itself is being removed from TripAdvisor.” The case generated another wave, even the Scottish parliament discussed the hostel’s conditions ”that generously, could be considered unsuitable, and, less generously, grim, Dickensian, like a Soviet gulag  or similar descriptions 4.”

Benevolence, hoax, permanently beta Any system established to help people and make fair profit can be gamed. It was in the news in 2015 that a non-existing Italian restaurant made the top of TripAdvisor ranking in a northern Italian small town, Moniga del Garda. Italia a Tavola, a leading on-line newspaper for information on food and wine made a hoax to prove the manipulability of rankings on the the portal. First, a profile was created for an imaginary restaurant, named La Scaletta, second fake reviews were generated by a number of conspirators. After receiving excellent reviews, the restaurant made the top spot in the list of the town’s restaurants. I am inclining to agree with those arguments, who feel the newspaper’s methods as ”unethical”. It is more efficient to work directly for a better world, than to try to pinpoint the negative sides of everything which exists. We should understand and accept that softwars are in the permanent beta state. ”Beta” originally referred to the final stage of software development immediately before the product was launched to the market, and a community of ”beta users” gave feedback. Nowadays, many product remained in this stage, and subject of continuous improvement.

Nobel and Oscar: the candidates and the winners

Nobel prize

One important signifier of reputation is the receipt of a prominent award or honor in one’s discipline or community. It is difficult to deny that Nobel prize can be identified with being ranked as #1. The Nobel prize takes its name from Alfred Nobel, a chemist, inventor, engineer, entrepreneur, and author who invented dynamite and held 355 patents during his lifetime. In his will, Alfred Nobel set aside his fortune to fund prizes for outstanding achievements in physics, chemistry, physiology or medicine, literature, and humanitarian or peace-related work. Since 1901, the Royal Swedish Academy for Sciences, the Karolinska Instituet, the Swedish Academy, and a committee elected by the Norwegian Parliament have been responsible for awarding the Nobel prizes, and in 1968, the Sveriges Riksbank established a prize in economic sciences that is awarded alongside the Nobel prizes. (There is a recurring gossip about the reason why there is no Nobel prize in mathematics. Different versions of the rumor claim that it is due to an eventual rivalry over a woman between Magnus Gösta Mittag-Leffle, the leading Swedish mathematician and Nobel. The rumor seems to be unjustified 29. To date, 590 Nobel Prizes have been awarded to 935 Nobel laureates.

The nomination and selection process The nomination and selection process for Nobel laureates takes over a year and involves dozens of reviews and expert consultations to determine the winner(s) in each prize category. In all categories but the Nobel Peace Prize, nominations are accepted by invitation only, and nomination forms are sent out in September of the year preceding the awarding of the Prize. Invitations for nominations are typically sent to selected professors at universities and to former Laureates, who have until January 31 of the following year to submit their nominations. The process typically generates about 250􀀀350 unique nominees, and consultation with experts proceeds for several months to assess the worthiness of each candidate. Over the summer, the Nobel Committee writes a report to be submitted in September to the institution responsible for awarding the particular Nobel Prize, and in October a winner is selected through majority vote and announced. Prizes are then awarded in December.

Literature prize: from betting to scandal In the last fifteen years or so, the big sports betting company, Ladbrokes has begun taking bets to the literature prizes in their on-line menu. So you can bet, who will win the Nobel Prize in Literature. (Only in literature, probably the others attract much less public attention.) Bookmakers at betting companies set fixed odds on all horses in a specific race, or on the result of soccer games. Betting became a big industry and even bigger business. The duty of a Nobel Prize bookmaker is particularly challenging. If you bet for sports events, you have a lot of data about past achievements, actual injuries, etc. Predictions based on finding patterns in data are not terribly bad. For the prize in literature, I am note sure. You should have a behavioral model of the committee. Betting for the Nobel prize somehow reflect people’s demand, and we could not exclude, that the selection committee does not totally neglect the public opinion. Well, we have a long list of unlikely non-winners (Tolstoy, Chekhov, Proust, Kafka, Virginia Woolf, Joyce, Nabokov).

In 2017 Kazuo Ishiguro was awarded for his ”novels of great emotional force, has uncovered the abyss beneath our illusory sense of connection with the world.” Margaret
Atwood, Ngugi Wa Thiong’o and Haruki Murakami had better odds, still Ishiguro’s win was approved by the community.
Is there any collusion between the selection and the betting process? I would have liked to bet that it isn’t. The committee receives annually about two hundred nominations from literary nobilities. Maybe ten-fifteen percent are first-time nominees. I might be wrong, that I believe that first-time nominees had initially a better chance than now. Rabindranath Tagore (1861-1941; NP: 1913), Sinclair Lewis (1885-1951; NP:1930, Pearl Buck (1892-1973; NP:1938, Bertrand Russell (1872-1970; NP:1950) and William Faulkner (1897-1962; NP:1949) were awarded Nobel Prize in Literature after being nominated in one year only. The ugly sex and leakage scandal 30, which led to the cancellation of the award in 2018 might have a cataclysmic consequences
for the Nobel prize in literature. We will see.

The illusion and manipulation of objectivity: Is there any gender bias? As noted above, 935 individuals have received Nobel Prizes since the Prize’s inception in 1901. Yet only 51 women have received the Prize (Marie Curie won twice), meaning that less than 6% of Nobel Prize winners were or are women. In the sciences, the chemist and science writer Magdolna Hargittai notes, that figure is even smaller, with just 19 women receiving the prize in physics, chemistry, or medicine or physiology since its inception. This abysmally low number has led some to suggest that the Nobel Prize exhibits a gender bias against female scientists. Certainly, there have been female scientists that have been overlooked for Nobel Prizes during the nomination and selection process, but it may be the case that the relatively low number of women who have been awarded the Nobel Prize is attributable more to implicit bias than explicit discrimination. Stereotypes discouraging women from pursuing education and careers in STEM fields have helped the number of women receiving doctoral degrees in physics, chemistry, and medicine relatively low in comparison to their male counterparts, and although these numbers are increasing, they remain lower than population statistics would suggest. Take physics as an example: according to the American Institute of Physics, in 1975 women earned just 5% of PhDs in physics, and they still earn only 18% of PhDs in physics. If less than one in five physics PhDs is awarded to a woman, the odds of being awarded a Nobel Prize in Physics has shrunk significantly for a woman. Even after PhDs are awarded, implicit bias manifests itself in the barriers that women face to hiring and publication, as studies show that women are more likely to be judged on superficial qualities like appearance and personal information than the quality of their scholarship. In terms of publication, women are less likely to be cited than men and their research is more likely to be attributed to men, and they are underrepresented in journal editorships. Both of these factors work against the likelihood that women will be invited to speak at conferences to present their findings or to be nominated for awards. Since the Nobel selection committees refuse to publish information about nominees until 50 years after the nomination has been submitted, it is difficult to assess the rate at which women are nominated for Nobel Prizes in STEM fields, but based on the state of gender representation in physics, chemistry, and medicine, there is a good chance that women are not being nominated at the same rate as men are. If we acknowledge the kinds of institutionalized biases that greet women at every step of their careers, it is less surprising that so few women have received Nobel Prizes in STEM fields. Although bias certainly exists, the nature of implicit bias is such that it is nearly impossible to pinpoint at which stage in a woman’s career her chances of receiving a Nobel Prize were diminished, and it may be the case that the bottleneck occurs well before names ever reach the Nobel Prize committees.

Ranking of artists

Péter Érdi: RANKING. The Unwritten Rules of the Social Game We All Play

Marek Claassen: Director of ArtFacts.Net

We introduced a quantitative method to measure how much an artist is embedded in the international art world. We start with long term relationships between artists and galleries or collections that represent them. These are very strong commitments that last very long. We count the number of countries and the number of collections and galleries. And then we look at solo and group shows. The more international artists a gallery or a museum has, the more its exhibitions value. Let’s say that we have an institution like Tate Modern where thousands of artists are collected. If you have a solo show there you get all the points from these artists and your rank will go up extremely. Biennials, group shows work like collections, their value is based on the artists whose works they show. So if there’s an Andy Warhol its value goes up a lot.

View original post 3 more words

Ranking of artists

Marek Claassen: Director of ArtFacts.Net

We introduced a quantitative method to measure how much an artist is embedded in the international art world. We start with long term relationships between artists and galleries or collections that represent them. These are very strong commitments that last very long. We count the number of countries and the number of collections and galleries. And then we look at solo and group shows. The more international artists a gallery or a museum has, the more its exhibitions value. Let’s say that we have an institution like Tate Modern where thousands of artists are collected. If you have a solo show there you get all the points from these artists and your rank will go up extremely. Biennials, group shows work like collections, their value is based on the artists whose works they show. So if there’s an Andy Warhol its value goes up a lot.

Top 100 Artists