There are the obvious injustices, from the standpoint of academic merit (i.e., actual faculty and student quality), in the overall rankings: numerous schools are ranked too low (Chicago at 6th, Berkeley at 11th, Texas at 15th, USC at 18th, Illinois at 26th, Emory at 32nd, Hastings at 39th, George Mason at 41st, Florida State at 56th, San Diego at 63rd, Chicago-Kent at 65th are just a few that catch my eye right away); numerous others--benefitting from the criteria that usually favor small, private schools--are ranked more highly than they deserve on the academic merits (Northwestern at 10th and Duke at 11th are two obvious ones at the high end).
In any case, grousing like this is old news, and is no doubt going on nonstop at faculty lounges around the country right now. Students trying to make sense of the information (and misinformation) provided by US News should review the methodology and its weaknesses in order to get the most out of the data provided.
What I'm particularly struck by is the "echo chamber" effect noted in my earlier posting: namely, that the academic reputation rank of at least the top schools is basically gravitating towards the typical overall US News rank of the school--a rank that is, itself, based in large part on that reputation! (The lawyer/judge reputation rank fluctuates meaninglessly year in and year out, mostly because the response rate is so poor: only 27% this year filled out the surveys!)
To check my sense that this is what's now happening, I looked at the reputation score of the top 15 schools in 2002, the reputation score in 2005, and then the overall US News rank of the school in the intervening period. The pattern, alas, is unmistakable.
In 2002, for example, Columbia's reputation rank was tied at 4-5. In the four intervening overall rankings, Columbia was always 4th. Lo and behold, in the year 2005, Columbia's academic reputation is now 4th, by itself (no tie).
In 2002, Chicago was tied at 4th in academic reputation; after four intervening years of an overall ranking of 6th, Chicago's academic reputation slipped to 5th.
In 2002, NYU's academic reputation rank was tied at 8-9 (with Virginia). In the four intervening years, NYU's overall rank was always 5th. By 2005, NYU's academic reputation rank had moved up to a new cluster, 6-8 (tied with Berkeley and Michigan).
In 2002, Duke's academic reputation was tied at 10-11 (with Penn), this coming on the heels of several years when Duke, overall, was ranked in the top ten. (Penn meanwhile had been ranked outside the top ten for a couple of years at this point.) From 2002-2005, Duke's overall rank was 10th only once, 12th twice, and 11th once. By 2005, Duke's academic reputation had slipped to the 11-13 cluster (tied with Cornell and Georgetown, and now behind Penn).
In 2002, Texas had an academic reputation tied at 12-15 (with Cornell, Georgetown, and Northwestern); in the four intervening overall rankings Texas was always ranked 15th. Lo and behold, in the year 2005, UT's academic reputation rank is now 15th, behind Cornell, Georgetown, and Northwestern.
The pattern is the same across the top 15 (with only one partial exception--Georgetown): schools consistently ranked higher (overall) than their academic reputation score saw their academic reputation score improve by 2005; schools consistently ranked lower (overall) than their academic reputation score saw their academic reputation score decline by 2005.
The problem here is that most of these changes are unrelated to anything other than the overall U.S. News ranking. Duke's faculty in 2005 is clearly stronger than its faculty in 2002, yet its reputation score declined (of course, it was too high in 2002); the reverse was true at NYU, yet its reputation score improved even as faculty left. Texas recruited leading senior faculty from Stanford, NYU, and Michigan (and improved the student body), and saw its academic reputation score decline in the same period.
The moral is clear: to improve your academic reputation according to U.S. News improve your U.S. News ranking! Improving your faculty or your student body doesn't matter.
If any law professor wants to see if this echo chamber effect extends more widely, please feel free to compile the data and send it to me.