I decided to write a non-fiction book with the title and subtitle RANKING – The reality, illusion and manipulation of objectivity. The book discusses the Hows and Whys of our love and fear of making ranks and being ranked through many real life examples to be viewed from three different angles (reality, illusion and manipulation) of objectivity. Ranking converts scientific theories to everyday’s experience by raising and answering such question as:
- Are college ranking lists objective?
- How to rank and rate states based on their fragility, corruption or even happiness?
- How to find the most relevant web pages?
- How to rank employees?
Life and society is really complex, consequently our message is not so simple such as ”Ranking is good!” or ”Ranking is bad!”. Since we permanently rank ourselves and others and are also being ranked, the message is twofold: how to prepare the possible most objective ranking and how to accept that ranking does not necessarily reflects our real values and achievements. The reader will understand our difficulties to navigate between objective and subjective and gets help to identify and modify her place in real and virtual communities by combining our human intelligence with computational techniques.
From RANKING to REPAIR! 
Very interesting topic. My compliments. It will be an interesting topic 🙂
LikeLike
Would perhaps things like order theory https://en.wikipedia.org/wiki/Order_theory, partial orders https://en.wikipedia.org/wiki/Partially_ordered_set and Hasse diagrams https://en.wikipedia.org/wiki/Hasse_diagram be (made) relevant?
LikeLike
To make it more popular, I was thinking of an example of ranking cars. Fully ordering them would give you the best, second best, etc. (called a “chain” in order theory). So if car A is faster and has better mileage than B, then A is better (higher in this chain). But if car C is faster than A but has worse mileage, and car D is slower but has better mileage than C and D are no longer “comparable” (a term from order theory). Then you have a graph (DAG), not a chain any more. It’s a more interesting kind of “ranking”.
LikeLike
LikeLike
Thank you! Very helpful!
LikeLike
If you prefer this: https://www.youtube.com/watch?v=i6oGytt0Hiw
LikeLike
Love the rabbinical and political science slant on the post.
What would happen if we had a ball and no one played?
LikeLike
I believe that no ranking is objective. Different objects/people can get different rankings based on what we are looking for. For example, in an organization, an employee who is ranked first in term of efficiency may be ranked last in terms of social interaction. The whole ranking system is based on our preferences, and in the real world, nobody or nothing is factually ‘better’ than the other from an objective perspective.
LikeLike
Deepti:
I will post a little remark to this comment
LikeLike
Best wishes for your project. I look forward to reading your analysis of Arrow’s theorem and related voting “paradoxes”.
LikeLike
Let me share a few thoughts on rankings. I tried to arrange them as a line of thoughts, but I am not certain about these thoughts
I consider everything from the point of view of its goal(s). Intelligence can’t present itself without visible and explicit goals. In turn, ranking should concern goals for those who are making (selling) it and should have value for those, who are using (buying) it for satisfying their own goals. Since value (money) is created, ranking may be able to sustain itself. Since value (money) is involved, ranking may be influenced by those who are to be ranked. Thus, the key question is how to keep the ranking “objective” in a sense that it can’t be influenced by the value (money) the users can make by exploiting it, but that value (money) increases with the number of users.
One point to make is that rankings have goals and rankings that can maintain themselves create values. Thus, they have cost functions in an explicit or implicit manner. The question is if a new ranking with a novel cost function can be estimated by means of existing rankings and if yes, then ranking can be personalized or not. In turn, rankings are closely related to recommender systems.
If we say that rankings are recommender systems then they should have predictive power. The value of the ranking depends stronly on the precision of the prediction it is making. In turn, we might have an objective measure for those rankings that can be seen as recommender systems.
Once I (and my colleagues) tried to develop a system [1,2] where rankers can express their opinion freely but can’t be punished for their opinion. It received some attention, but — as far as I know — it was never used. It might still be relevant for some rankings, I don’t know.
[1] Farkas, C., Ziegler, G., Meretei, A., & Lőrincz, A. (2002, November). Anonymity and accountability in self-organizing electronic communities. In Proceedings of the 2002 ACM workshop on Privacy in the Electronic Society (pp. 81-90). ACM.
[2] Ziegler, G., Farkas, C., & Lőrincz, A. (2006). A framework for anonymous but accountable self-organizing communities. Information and Software Technology, 48(8), 726-744.
LikeLike
I fully acknowledge the concept of having a goal to all intellectual activity, including ranking, and also that the ranking itself might influence the behavior of the ranked. It is somehow similar (or at least it resembles) to the observer effect in Physics: the fact that simply observing a situation or phenomenon necessarily changes that phenomenon. In physics, this is often the result of instruments that, by necessity, alter the state of what they measure in some manner. To this respect ranking is an instrument of observation.
See https://en.wikipedia.org/wiki/Observer_effect_(physics)
LikeLike
It seems as you have some original ideas/views on this topic, rather than merely synthesizing the available literature.? Anyhow, this topic and especially your outline sounds very exciting and I am looking for your book to be published soon. This intro already provokes questions about what we are humans. Is an/the objective measure as you write (as I understood), promises the possibility of a (sufficiently) generalized (trustworthy) way of ranking? This sounds tough considering the diversity of human abilities and also that of the expectations that those should match. Let’s see the solution(s)!
LikeLike
What a great topic! Please consider me as a reviewer for your book!
LikeLike
Great and important topic. It is indeed time dig deeper in the impact of rankings that greatly influence decisions. Looking forward to read more.
LikeLike
Like heuristics, rankings can be a useful (though limited) concept to provide information in a way compatible with our cognitive biases and limitations. However, one can wonder whether the increasing availability of powerful information presentation and manipulation tools has made them obsolete.
LikeLike
in the world of big data in which we are presently saturated and with the endless competition to be in the top rankings of anything — from credit cards and universities to twitter posts and Instagram — this is definitely an important topic. particularly as we see the manipulation of rankings and proliferation of fake-news and fake-data. Well done! I look forward to reading more.
LikeLike
A deep, multi-layered topic. Rankings, put out of context, can be highly misleading. Even at a basic level, the pressure of being ‘ranked’ is not suitable for all; for example forcing a generalised ranking system on all children in a classroom will bias the class rankings anyway!
One can go on and on, and therefore this endeavour towards a book seems very apt indeed!
Shall wait eagerly for the read. My best wishes Peter.
LikeLike
Best wishes for this interesting and important topic! Social animals also have some ranking. It would be interesting to see some outlook and connection to this area, as well.
LikeLike
Pista:
Thanks!
Could you find the follow button! I will post something bout |1.3. Social ranking in animal and human societies”. Kind regards, Peter
LikeLike
When learning about the book idea, I have been thrilled. The topic is extremely interesting, with rankings going wild at several places (universities, journal impact, persons, …) It would be more than timely to set this into perspective and to unvover some of the secrets which lead people to act in this way – albeit scientists know the problem of linearization of multicriterial objectives… I will love to read this book.
LikeLike
Thank you, Barbara! Hope you will follow the site, and will make more comments!
LikeLike
Well, I think that rankings are important cognitive tools. Generally, they seem to be useful if they cover not too many items (e.g. neighbours in the street, immediate colleagues at work, neighbouring towns / villages, few car labels, etc.) – probably not more than a few tens at most. These rankings may have multiple components which may work together as a partial order. In institutional environments rankings get extended and rationally processed and turned into complete orders. However, I think that such large scale institutionalized rankings do not work intuitively for most people. One possible impact of the book could be to make large scale rankings more intuitively usable for people in general.
Given that intuitively usable smaller scale rankings are partial orders, it is possible that the social ranking algorithms correspond to rationalized recovery of the transformation of ‘natural’ intuitive rankings into complete orders.
It would be also interesting to see how large scale complete order rankings become operationalised in cognitive terms by converting them into small scale partial order rankings of selected items from the large scale rank (e.g. picking of a university or picking of a job).
LikeLike
Congratulations on your decision to challenge to this difficult but important subject!
LikeLike
Thank you Ichiro. It is fun!
LikeLike
I’ve been applying to grad school the last couple months; Dr Erdi asked me to talk a little bit (though, I might have written more than a little bit, so this might need to be broken into two comments) about that experience and how I used rankings through the process.
I’m applying to masters programs in public policy/affairs. Going into the selection process, I knew relatively little about what programs were good, or even, which programs existed; I knew most big schools had something, but didn’t really have any idea of how to begin evaluating them against one other. So, naturally, I turned to US News & World Report’s needlessly encyclopedic school rankings. This is, more or less, where I started; I knew two schools I would be applying to for sure, but for the rest of the list, I drew fairly heavily on the USNWR’s listings, at least initially. I’d taken the GRE the previous year and knew my scores were competitive for top- and high-ranked programs, so I just started at the top.
By the field’s nature, public policy schools tend to be fairly establishmentarian, but broadly speaking, the good schools can further classified into two types. First are the named-for-a-Founding-Father, beltway/Ivy League outfits that, while fine programs, largely function to create future congressional staffers and lobbyists. The second type of school I generally saw was the well-rounded, often-Midwestern state school with a strong emphasis on both quantitative and written work; a school with a more liberal, open-ended approach to education. I want to work on a fairly niche topic through an interdisciplinary lens, so this is the style of school that I was primarily drawn to (5 of the 6 schools I eventually chose were from this second group).
So, at the point where I was deciding the exact programs I would apply to, I knew a couple of things: first, I’d be applying to 6 schools because it just seemed like a reasonable number; second I wanted to avoid the especially traditional East Coast schools (barring one exception that went out of its way to avoid the pitfalls of other schools of its type); last, I knew two schools that were on the list, one in the town I was already living in, and one as a safety school.
LikeLike
From there, I worked down the ranking, looking through probably the top 50 programs. Something like half were eliminated for not offering the more liberal approach I outlined earlier, and about half of those left were eliminated based on either location, or program size, leaving 10 or 12 left to choose from. Of those left, I narrowed it down by faculty/research areas and the ability to take classes outside of the program, as well as less academic concerns like location. All told, my final list came out as University of Michigan, University of Chicago, Wisconsin-Madison, Ohio State, Duke, and Indiana University.
Regarding rankings more generally, as abstract or cognitive devices, I would echo a lot of the comments that have already been made. I think generally, we’re skeptical of the objective capacity for most heuristics, but I d2on’t we recognize the degree to which rankings fall under that umbrella. I think on some level we do recognize it, especially with things like college rankings; however, as soon as we say that such rankings aren’t exactly, perfectly objective, we seem content with our level of analysis (say, listing but not ranking 40
colleges that ‘change lives,’ so long as we mention that there’s no ‘best college’ for everyone). I think this book definitely has some fertile ground ahead of it, especially with regard to the way that ideology and methodology inform how we make rankings, and how we then approach those result as ‘objective’.
For my school search, I knew that the rankings weren’t ‘objectively right,’ per se, but I knew that the way they were calculated more or less cohered to how I would go about evaluating them, and made corrections along the way to adjust for the way that the rankings and my own evaluations might deviate from one another. In other situations, I don’t know that I make that sort of consideration, and I don’t think others make such considerations that regularly either. But either way, I think my initial instinct (that is, checking for rankings and moving from there) is becoming a more and more preferred method of operating, so I’m very excited to see where this research goes.
LikeLike
Thank you Brian! it is great to see how to use a ranked list in your real life!!
LikeLike
I am truly looking forward to seeing -and eager to read- this lats Peter’s effort. The topic is fascinating, the content truly up-to date, also given all number-less applications where “ranking” takes part. I am sure I will learn a lot from this book as “ranking” is a silent word permeating my life…
LikeLike
Your topic (ranking) is one of the most irritating insidious practices of human beings! Some how we have built “rank” into just about every aspect of our existence. We often use hierarchical models for decision making in situations where it is totally inappropriate. I spent the first fifty years of my life trapped in the misconception that decisions could be made using “best” to “poorest”. The fallacy of that model became apparent to me when I had to fight with other academicians regarding the “worth” or appropriateness of specific educational levels when they insisted in using the hierarchical model in talking about education that lead to occupations. I could not respond to the question “Which is better for me…railroad engineer or research engineer?” using the hierarchical model because that hierarchical model for choosing a major or program just did not take into account the needs and aspirations of the person. I discovered that I could rotate the hierarchy into a spectrum of engineering field occupations and convert their question into “What would you like to do for a living?”. I created a video to use in advising that shows how to use the spectrum approach to explain educational expectations (outcomes). I suspect that there are many decisions people need to make that would result in a “better” outcome if they used a spectrum of options as opposed to a hierarchy of options.
LikeLiked by 1 person