The US News law school (and other graduate program) rankings will be officially released tomorrow (and I'll have a few additional comments then on some of the particulars of the results for law and for the humanities and social science programs), but it's worth noting that US News changed the way it calculates the "median" LSAT and GPA which is what it factors in to the overall ranking (the magazine has only printed, this year and prior years, the 25th/75th data). The American Bar Association collects the 25th/75th data from the schools, and, as I've noted before, schools that might fudge with US News are far less likely to fudge with the ABA, since the ABA also plays an accreditation role. This means that when private schools, in particular, report the median to US News, there is simply no way to verify the accuracy of the numbers being reported.
This year, for the first time, US News "calculated" the median, and ignored the self-reported medians. Of course, what they "calculated" wasn't really a median, but simply the mid-point between the verifiable 25th and 75th percentile data. So, to take an example, our fall 2004 median LSAT was 165, but the 25th/75th spread was 162 to 167, meaning that the figure US News employed for ranking purposes was not 165, but 164.5.
My own opinion is that this is a constructive change, from the standpoint of the reliability of the underlying data. It's true, as I've written elsewhere, that there's still a real problem with the accuracy of a lot of the data that goes in to the ranking, but at least in this case they've increased reliability. The median data that goes in to the ranking is no longer manipulable through fraud. (It's still the case that the use of median data advantages smaller schools: e.g., Washington & Lee will have had the same mid-point LSAT as Texas for rankings purposes, though Texas will have recruited at least three times as many students with those credentials.)
On the other hand--and this is how I first heard about this from colleagues elsewhere (and then confirmed it with Bob Morse, the Director of Data Research for the rankings issues at US News)--many are worried that this is going to negatively impact "alternative admissions" programs (affirmative action, preferences for alumni children, and so on), since these programs typically involve admitting students with lower numerical credentials, which is far more likely to affect the 25th percentile scores than the median. That risk is real, though it may require law schools to show a little fortitude in their commitment to their institutional missions, and not cede all their decision-making to US News ranking criteria.
Here, in any event, is what Mr. Morse had to say about these changes (and he gave permission to share his comments); I had forwarded to him some e-mails from other legal academics worried about the implications of the change and wondering whether it had been made. Mr. Morse wrote:
[I]t's true that we are using a new way of calculating the midpoint LSAT and GPA score. Our intent wasn't to hurt minorities or change law school admission office behavior or set up new incentives or disincentives. It was to use verifiable ABA required data and to prevent misreporting of the median. All schools were treated the same. We had heard reports that schools were using a different calculation system for the median than then they did for their ABA 25th and 75th. If it become clear that the unintended consequences are highly negative, we will surely reconsider this move.
Mr. Morse subsequently forwarded the following interesting information about the impact of this change:
More factual information to circulate if you want. To us this means in bold [below], that the difference was very very little between calculated midpoint for almost 90% of the schools in terms of LSAT. This also means for the vast % of schools this had virtually no impact on their rankings. I'm not commenting on how this will or will not change behavior. You can cirulate this if you want, too.
There were 178 schools that reported medians for both the LSAT and
undergraduate GPA for fall 2004 entering FT class. Not all of these were ranked schools.
The calculated median [i.e., midpoint] undergraduate GPA was, on average, about .02 less than the reported median.
The calculated LSAT was, on average, .25 less than the reported median.
For the LSAT, 27.5% of the schools had a reported median that was the same as their calculated median. 69.7% had a difference that ranged from -.5 to +.5. 88.8% had a difference that ranged from -1 to +1.
For the undergraduate GPA, 5.1% had the same calculated and reported median. 80.3% had a difference that ranged from -.05 to +.05. 95.5% had a
difference than ranged from -.1 to +.1.
The undergraduate GPA didn't have as many with the same calculated and
reported medians probably due to the larger scale. We also don't know if some schools just reported it to 1 decimal place while other schools reported it to 2. On the LSAT, there were 4 schools that had a difference greater than or equal to 2 LSAT points.
UPDATE: William Henderson (Law, Indiana-Bloomington) makes some good points in response:
Thanks for researching this US News methodology issue.
With all due respect, the new change is not constructive. Your point on fraud is well-taken. For the last several years, my colleague, [name omitted], waged a fierce battle with the director of the ABA Legal Education section to make sure the ABA continued to collect the 50th percentile. His point was simple -- if you don't have this data to root out false numbers reported to US News, then US News will gravitate toward the 25th / 75th percentiles since they have greater assurances that this data is accurate. Since the ABA dropped the median as part of the collection process, [my colleague] has had repeated conversations with Bob Morse asking him to stick to the 50th percentile. Finally the specter of gaming drove Morse to the very outcome [my colleague] predicted. The solution here is simple. The ABA should collect 25th, 50th, and 75th percentile data, and US News should return to is former practice.
Morse's observation that the new averaging technique produced numbers very similar to the reported medians is NOT a basis for relief. Admissions directors, much to their chagrin, are operating under a new set of rules. The 25th and 75th become the key benchmarks, making the whole admissions process even more LSAT-centric than it has been to date. Your comments on "fortitude" are not realistic in light of the complete lack of fortitude the current rankings game has produced. When you have a systemic temptation to err on the side of higher LSATs, you can't count on 180 law schools to stay on the fortitude wagon. As some fall off, others follow. This is a big collective action problem.
By the way, my deep-seated skepticism of the LSAT as the be-all and end-all of the admissions process is informed by empirical research. This article on the LSAT was published in the Texas Law Review last year. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=465381 Your blog is very influential--and most of the time I agree with you. But I think your endorsement of this change in the US News methodology is a mistake.
Recent Comments