Brian Bruya, who teaches philosophy at Eastern Michigan University and earned his Ph.D. at the University of Hawaii (specializing in Chinese philosophy), has penned a purported critique of the PGR, which an actual journal, Metaphilosophy, published (readers will see in a moment why this is surprising). Why do I mention these facts about the author? Because the author has reasons to be hostile to the PGR--he plainly feels he and his friends in the profession are undervalued because of the PGR--it should not be wholly surprising that the critique the author produces involves mistakes, shoddy arguments, and fabrications.
This is no longer my headache, but since the article ignores the participation of the Advisory Board in producing the report for the last 15 years, and continually personalizes his critique to me, let me note the following:
1. Bruya asserts (674), falsely, that the "Metaphysics & Epistemology" (M&E) category in the PGR includes 15 sub-specialties, more than "Value Theory" and "History of Philosophy" together. In fact, the M&E category has only 7 specialties listed, compared to 6 for Value Theory and 9 for History of Philosophy. In fact, there are 8 specialties under "Philosophy of the Sciences, Mathematics, and Logic," which is the fourth major sub-division of the field the PGR has used for a long time; these four divisions correspond quite well to the areas represented by about 95% of philosophers in the Anglophone world (most of the rest are picked up by the areas that don't fit neatly into any of the others, because they cross divisions, like Feminist Philosophy or Philosophy of Race). Buried in an appendix at the end, Bruya finally acknowledges conflating the divisions, with the explanation that, "It is uncontroversial that many of the specialties of M&E, philosophy of mathematics, and logic are core specialties of Analytic philosophy" (686). This is, of course, revealing about Bruya's biases, and his lack of understanding of "analytic" philosophy (he might talk to some philosophers of physics and biology to find out what they think of a lot of, say, contemporary metaphysics). But the mistake vitiates in its entirety the "analysis" on pages 668-678, where the statistical analysis purporting to show "bias" in the evaluator pool depends entirely on his massive inflation of the M&E category.
2. Snowball or chain-sampling is a perfectly appropriate method of sampling when what you want is a kind of "insider's" knowledge. Bruya's argument against it is silly:
The reason [snowball sampling] is used is as an expedient way to access a hidden population, such as social deviants (drug users, pimps, and the like), populations with very rare characteristics (such as people with rare diseases, interests, or associations), or subsets of populations associated in idiosyncratic ways (such as networks of friendships)....Philosophers are neither social deviants nor difficult to find, as every philosophy program's faculty list is public information. (660-661)
It does occur to Bruya that perhaps what is wanted is expert and well-informed opinions. Bruya comments: "the very earning of a Ph.D. is the academic standard for expertise" (661). If that's one's view of expertise, then the PGR will not make much sense. Later in the article, however, Bruya acknowledges that,
We want experts to provide their opinions when expertise is required for a sound assessment, and we would not insist on getting a representative sample of all such experts. We see this all the time in academia. We have PhD committees, tenure review committees, grant committees, and so on, which are formed for the purpose of providing expert evaluation. And for none of these do we insist on getting a representative sample. (667)
Given this admission, one might wonder what all the fuss is about? Bruya again points to "bias," but as we've seen (#1, above) the central and allegedly data-driven analysis of "bias" in the evaluator pool involves a straight up fabrication about the areas of philosophy represented by the evaluators.
3. In an unrelated effort to show "bias," Bruya asserts that there is a category of philosophers who are "Methodological Continentalists" (665-666) which would encompass programs like DePaul, Duquesne, and Emory (666). SPEP folks have long maintained, of course, that they are insulated from normal standards of philosophical scholarship because there's something putatively distinct about the work they do that makes such standards irrelevant. Bruya is entitled to endorse that myth. But what is, again, pure fabrication is to assert that "Leiter refers to this brand of philosophy in his own published work," noting the introduction to The Oxford Handbook of Continental Philosophy I wrote with Michael Rosen (666). Bruya gives no page reference, because there is none that would show that we recognize something called "Methodological Continentalists" represented by departments like DePaul, Duquesne, and Emory. How such a naked fabrication got through the peer review process is, again, mysterious.
4. Bruya repeatedly misrepresents Kieran Healy's research about the PGR; readers interested in Prof. Healy's views can start here.
5. Bruya's main methodological suggestion (681 ff.) is to aggregate scores in the specialty areas for overall rankings. The Advisory Board discussed this in past years. Since there is no way to assign weights to the specialty areas that would not be hugely controversial and indefensible, the PGR has never adopted such an approach. Bruya has no real solution to the problem (though one may rest assured Chinese Philosophy will count for more!).
I predict, with confidence, that no changes to the PGR methodology are likely to result from this very confused critique.
I also trust the journal Metaphilosophy will withdraw the article in its entirety given the fabrications, and subject a revised article to a more serious peer-review process, to insure the final version is not so obviously shoddy.
ADDENDUM: I suppose I should say a word about Bruya's defamatory claim that "one cannot help but wonder whether the PGR's hidden biases are based in sexism, racism, ethnocentrism and xenophobia" (679). Obviously, Brit Brogaard was motivated to join me as co-editor of the last PGR by the opportunity to help propagate "sexism, racism, ethnocentrism and xenophobia." So were the women, African-Americans, and Hispanics on the Advisory Board; and the dozens of women, African-Americans, and Hispanics who participated as evaluators. An alert reader might notice that Bruya motivates this smear first through his fabrication of a huge M&E category (see #1, above) and second through his unmotivated claim that "a program that was both complete and perfectly balanced...would have a total of thirty-three faculty members," one for each of the 33 specialties ranked in the PGR. Why should any program aspire to represent every specialty in the PGR? Bruya never explains. But, of course, if you start with that assumption, then it looks like Chinese philosophy, not to mention philosophy of religion and medieval philosophy and philosophy of art, get shortchanged. It never occurs to Bruya that this might be a fact about the profession that the PGR records, rather than creates (in fact, during the PGR's lifetime, the range of philosophical areas well-represented in the profession has increased, and that's been reflected in the PGR).
DEC. 15 UPDATE: After this update, I am not going to spend any more time responding to Brian Bruya, who in 14 single-spaced pages tries to create the appearance that he is responding to the criticisms, above. In some cases, it is obvious that he has no response, posturing to one side. I'll just make seven quick points:
1. Bruya claims that the "reason I wrote the critique of the PGR is that I see a group of very intelligent and apparently well-meaning people involved in an influential publication that ultimately locks out" non-Western philosophy. These are the same people whom Bruya smears in the article as having hidden biases deriving from "sexism, racism, ethnocentrism and xenophobia."
2. I noted that "the article ignores the participation of the Advisory Board in producing the report for the last 15 years, and continually personalizes his critique to me." Bruya drops everything after the comma, and then responds, irrelevantly, that his article mentions the Advisory Board in a couple of places. Anyone who bothers to read Bruya's article can see very clearly that throughout he treats the PGR, and all the bad things about it, as due to me, personally. He ignored the fact that it has been, for a long time, a collective effort. Thus, he does, indeed, "personalize" the critique.
3. Bruya's M&E category is a fabrication, and as I noted, he does finally acknowledge it to be so in an appendix at the end of the paper. The fabricated category allows him to claim that there are more M&E categories than value and history categories. It also vitiates his statistical claims, which trade on the huge M&E category he created. There's no doubt that, as Kieran Healy has shown, that metaphysics, epistemology, philosophy of language, and philosophy of mind are high prestige areas in Anglophone philosophy; the PGR records that fact, it didn't create it (a point on which Bruya continues to be silent). That does not justify Bruya's fabrication, which he still doesn't defend or even explain. (I did laugh out loud when the guy who smeared me and the PGR participants as infected by "sexism, racism" etc. worries that it might be ad hominem to note that he doesn't seem to understand what "Analytic philosophy" is.)
4. I noted that the 33 M&E specialties represent 95% of what philosophers do. Bruya repeats the non-sequitur that an APA survey (of only participants in one Eastern Divison meeting, the Eastern Division being the most atypical of the three Divisions) would have resulted in a different distribution of specialties. Many (most? I haven't checked them all) of the specialties that Bruya labels "other" are, in fact, part of existing specialty categories. If my 95% estimate is wrong, we still haven't seen the evidence.
5. Bruya manages never to respond to the points, above, about expertise, though assures us that he thinks his argument was a good one. Readers can decide for themselves.
6. Bruya repeats the falsehood that Leiter & Rosen endorse his version of Methodological Continentalists, as exemplified by programs like Emory, Oregon, and Duquesne. He now produces a quote which shows nothing of the kind.
7. Bruya thinks an alternative way of producing an overall ranking is to simply "sum" all the specialty scores. I confess it never occurred to me or anyone on the Advisory Board that this could possibly be a sensible way to proceed. Readers may decide for themselves whether Bruya is persuasive on this point.
Bruya, I suppose, is counting on the fact that the appearance of a response, complete with school marmish posturing, will suffice, and that readers won't examine his confused paper carefully enough to see what's really going on. Maybe he is right. Nonetheless, I continue to predict with confidence that Brit Brogaard and a new Advisory Board will not adopt any of the suggestions offered by Bruya because she is smart and will appoint a smart Advisory Board, and it's easy enough for any smart person without an axe to grind to see that the criticisms are mostly confused or tendentious (not all, some are familiar, but not significant). I am also hopeful that Metaphilosophy, which allegedly has a peer review process, will withdraw the article in light of the fabrications and defamatory smears.
ANOTHER: Unfortunately for Prof. Bruya, the philosopher of physics David Wallace did examine the paper carefully, and his conclusions are damning.