The QS University Rankings are a fraud on the public

An untenured philosopher at a religiously-affiliated university in the American South writes:

I’ve noticed that the (rather shady!) QS rankings have appeared on your blog several times – and thought it might be interesting to briefly share my experience as a QS ‘evaluator’.

One summer not all that long ago, I was short on cash and signed up for a paid survey site.  Generally, this involved filling out online surveys about cell phones, toilet paper, and frozen vegetables for a reward of between $1 and $5.  One of the stranger surveys that came up was something called the QS Rankings. I assumed that I was selected to participate because of my educational background and employment (PhD in philosophy, employed full time by a university), and completed the survey without much thought.

Here is how the survey went.  First, I was asked to choose my discipline from a list of about 10 options as well as my continent and country; I chose ‘Humanities’, ‘North America’, and ‘United States’.  Next, I was asked to type in the five best universities in the US for the study of humanities; then I was asked to type in up to 15 top universities for the study of humanities in North America other than the five I put in before.  Finally, I was asked about my ‘sub-discipline’; I was asked to type in five top universities in the world for the study of philosophy.

The survey seemed awfully suspicious at the time, and I did not imagine that the results could have any sort of serious impact. Of course – I have not done much investigation into the QS rankings, and I would imagine that they use criteria other than paid surveys – but ranking universities by using a paid survey site seems to be poor methodology, for a number of reasons!  

So the QS rankings are being completed by people who happen to sign up to be paid to complete surveys!  (At least this individual was actually a philosopher!)  I'd be interested to hear from others who may have been surveyed by QS, and if so, how.  If they are really doing something as suspect as trolling fee-based survey sites for respondents, then this makes me wonder about the data bases they use for the purportedly 'objective' measures like citations and H-indices.  Why doesn't QS disclose the faculty lists they use?

UPDATE:  A philosopher in the Middle East writes:

As you mentioned you'd be interested to hear about how the QS company does the surveys, I thought I email you.

I received an email (actually, 12 emails consecutively) on April 16 this year asking me to register and complete the survey. I didn't do it, as it looked dubious. First, it came from a person from a "petroleum and minerals" university in Saudi Arabia, which has nothing to do with humanities whatsoever (it has no such fields of study): http://www.kfupm.edu.sa/default.aspx

Second, I could not find any info about the person who sent me the email. I googled his name, email, affiliation -- no result.

Below is the complete message.

 ---------------

Dear Researcher, 

Sub: Requesting your time for participating in the QS World Rankings Survey 2013.

The opinion of academics forms the very core of the QS World University RankingsR - the most widely circulated of the world rankings. As a leading

academic, and someone who has been associated with King Fahd University of Petroleum & Minerals (KFUPM), I am writing to request a few minutes of your

precious time to have your say in the 2013 Academic Peer Review Survey.

You will receive a message from QS providing you the link to complete the survey.  If you have not received the survey, then I would like to request

you to please visit the link:

<http://qsnetwork.qualtrics.com/SE/?SID=SV_5yB0wOlsG0d0IVT>

http://qsnetwork.qualtrics.com/SE/?SID=SV_5yB0wOlsG0d0IVT and kindly

register.

Thank you for your time and please contact me if you have any questions.

Nooruddin Ghalib

Administrative Assistant to the Rector

King Fahd University of Petroleum & Minerals (KFUPM)

Dhahran 31261, Saudi Arabia

AND MORE:  A law and philosophy scholar, not presently in a tenure-track position, writes:

A while ago (more than a year ago, but I can't remember for sure how long ago) I got an email like the one your correspondent got, about surveys of academic programs.  I _think_mine was about philosophy, but it might have been law.  I started filling it out, but pretty soon gave up- it was taking too long, and, given the odd way it was set up, I didn't really feel like I could answer well.  As in the email you got, it asked for names of departments, didn't give lists of schools (let alone of faculty), and so on.  It was obviously going to lean heavily towards names people could easily recall.  And, while I think I'm competent to contribute to such surveys in some areas, this one and in this
style left me pretty unsure that I was contributing anything, so I didn't finish it.  I didn't think much of it at the time, but now I'm pretty sure it was the QS rankings.  Apparently they more or less randomly email people and ask them to do a really poorly crafted survey, among other things!

UPDATE: QS's Head of Public Relations replies here.