The form of his purported reply to Wallace will be familiar to readers of his last attempt to defend himself. Wallace has already replied, in two parts, excerpted below:
1) It is deeply disturbing that Prof. Bruya doubles down on his relegation of his methodology to an appendix, and his appeal to a footnote as a justification. You just *cannot do this* in serious, quantitative scholarship. You have to make it explicit, up-front, crystal clear, what your methodology is – *especially* when you are using categories acquired from your critical target and rearranging them. Prof. Bruya says that this is “perfectly reasonable to me”. It is not “perfectly reasonable” by the standards of any quantitative-science journal I have ever read. (I welcome counter-examples.) In the absence of such, I continue to think that Bruya’s coyness about this methodology (and his coyness about his methodology on evaluators’ institutions, which he has not yet addressed) is inappropriate as a matter of scholarship.
2) Prof. Bruya resorts to the very worrying methodology of saying that my criticisms are only valid insofar as they also apply to the PGR. (He goes as far as to italicise the points where he feels a double standard is applied.) *This is irrelevant.* Either the reasoning in his paper is valid, or it isn’t. You can’t defend shoddy science by pointing out that others’ science is also shoddy. It’s fine to impugn my motives and object that I only spend time on criticisms of the PGR. (The truth is that the PGR’s defenders don’t make clearly-false mathematical claims, but we can dispute that.) But the question of what I bother to spend my time criticising has nothing whatever to do with whether a given criticism is valid. Does Prof. Bruya think his arguments are valid, or not? If so, let him defend them. If not, it’s irrelevant whether other arguments are also invalid.
3) Prof. Bruya notes that I have not addressed all of the objections he raises. His criticism predates the extension of my criticism to cover his conflation of two categories of institutional association (addressed only in a graph x-axis and never in the text). I didn’t engage with the main additional point he raises (regression analysis of PGR-vs-all-subject-aggregate mark) because I lack the data. If Prof. Bruya is prepared to email me his data-set (david.wallace@balliol.ox.ac.uk) I’d be very happy to play with it, with the proviso that I have a baby turning up in the next few days!
...
On reflection I think I spoke too soon when I said that I couldn’t engage with Prof. Bruya’s regression analysis without the data. It also appears to have a severe methodological flaw, though I don’t think this is as serious a point as others I’ve raised.
Prof. Bruya calculates the difference, for each US institution, between (i) the PGR rank and (ii) the “Bruya rank”, i.e. the equally-weighted sum of all speciality marks. He then runs a linear regression analysis to determine the dependence of this difference on the equally-weighted sums of speciality ranking in four areas: History, Value, “M&E” (being Prof. Bruya’s combination of the PGR categories of M&E and Science) and Other.
The problem is that these are very strongly correlated variables: generally speaking, departments with high PGR ranks in one category have high PGR ranks in others, and running regression analyses on very strongly correlated variables has to be done with great care.
To illustrate: suppose I’m interested in whether certain things about US voters (let’s say, their salary) can be predicted by their voting patterns. If one of my regression variables is “usually votes Republican in Senate elections” and another is “usually votes Republican in House elections”, then it’s very difficult to determine the relative contributions of those two variables to a voter’s salary, since each of them is a very good predictor of the other.
You can work around this with a sufficiently large dataset, and there are statistical tests that can be done to ascertain whether a result is still significant even given a high correlation, but Prof. Bruya does not mention any such test.
Recent Comments