David Hilbert (Illinois/Chicago) writes:
You're probably sick of NRC email but this seemed interesting enough to be worth sharing. I've been playing around with the NRC spreadsheet trying to understand the basis of our ranking. There's two interesting features of the R-rating that I haven't seen discussed yet (although perhaps I missed it).
I was interested in the coefficients used in constructing the rankings. If you look at the variables tab in NRC spreadsheet they give the plus and minus one standard deviation for the coefficients on each of the variables used in both the R- and S-ratings (for the discipline of interest). The coefficients in the S-rating are pretty much what one would expect to emerge from a survey of faculty with the publications variable receiving the most weight. One variable has a negative coefficient, the time to degree variable, which makes sense since larger numbers here are widely thought to show something bad about a program.
If one looks at the coefficient distributions for the R-rating system on the other hand there are two things that immediately stand out. First, there are several variables for which both the plus and minus one standard deviation coefficients are negative. That means higher scores on these variables are likely to have lowered a program's ranking (both at the 5th and 95th percentile). If one looks at percent female faculty, for example, the coefficients are -0.062 and -0.034. The negative signs on these coefficients means that having a higher percentage of female faculty members will tend to lower a department's R-rating. Given the way these coefficients were determined this makes sense if those departments that were higher in the sample representational ranking (that is not reported in the data) tended to have a lower percentage of female faculty. There were five variables for which the plus and minus 1 SD were negative: percent faculty interdisciplinary, percent non-Asian minority faculty, percent female faculty, percent first year students with external funding, and percent completing within six years. Doing better in any of these areas will tend to lower a program's position within the R-ranking. It's particularly striking that the two equal opportunity factors ended up hurting rather than helping in this ranking system. The problem here is that the one that Ned Block pointed out prospectively, the failure to distinguish between signs of quality and determinants of quality.
The second thing I was struck by was the low weights given to two out of the three variables relating to faculty quality in the R-ranking. Both the productivity measure and the percent faculty with grants count for less than does, for example, average student GRE or the average number of PhDs produced per year. If the R-rating were to be taken seriously and the coefficients held fixed then programs would have more effect on their rating by granting more PhDs than by increasing faculty productivity.
Since I imagine others have, by now, had a chance to digest the data, I invite other signed comments on the NRC data. I am particularly curious to hear how administrators at your school are responding to the NRC report. The University of Chicago appears to have been the only major research university to not issue a press release touting its performance, because of the devastating analysis of the value of the results by the statistician Stephen Stigler here. (Some of Stigler's points about the way the R-Ranking was constructed based on only partial reputational surveys may bear on the odd bias noted by Professor Hilbert.)
