Leiter Reports: A Philosophy Blog

Bruyahaha update: Brian Bruya published a purportedly "scholarly" article about the PGR...

...in a heretofore obscure journal called Metaphilosophy.  The article was incompetent, as I first noticed and as David Wallace (Oxford) confirmed in detail.  There was also, of course, the little matter of the article's defamation of those who produced the PGR--me and Brit, as well as the Advisory Board--as really being racists and sexists.

I have now had a correspondence with the editor of this allegedly "scholarly" journal--Prof. Marsoobian--and, to his credit, he did commission additional referee reports on Prof. Bruya's article in the wake of the criticisms.  Prof. Marsoobian declined to disclose either the new referees (recall that he also chose the initial referees, so one does wonder) or even an anonymized version of their reports--I can not blame him for the former, but think the latter revealing:  why not let skeptics see the assessment by alleged professionals?  Despite this secrecy, the best the new referees could say about Prof. Bruya's article was, according to Prof. Marsoobian, this:  "the methodology in the article is not so seriously flawed that it would warrant retraction."   It is good to know that the referees concur with Prof. Wallace that Bruya's article is "seriously flawed," as it obviously is.  But according to Prof. Marsoobian, the referees did not defend the article or its methodology or its analysis; all they could say was that it was "not so seriously flawed that it would warrant retraction."  What are the standards for retraction for incompetent work?  Prof. Marsoobian did not say.

Metaphilosophy will publish Prof. Wallace's critique.  As to the defamation, that issue may not, it appears, be resolved in the journal, though the matter is still under discussion with Prof. Masroobian and Wiley-Blackwell and counsel.

The latest from David Wallace on the Bruya mess

Prof. Wallace has updated his critique here.   Section 6, which was added in the second version of his critique, also raises, as Wallace notes, an "academic malpractice" issue.  Wallace also offers a useful summary of his take on the issue in light of Bruya's purported response:

Bruyahaha Update #2: guess what, Bruya tries to respond to David Wallace

The form of his purported reply to Wallace will be familiar to readers of his last attempt to defend himself.  Wallace has already replied, in two parts, excerpted below:

1) It is deeply disturbing that Prof. Bruya doubles down on his relegation of his methodology to an appendix, and his appeal to a footnote as a justification. You just *cannot do this* in serious, quantitative scholarship. You have to make it explicit, up-front, crystal clear, what your methodology is – *especially* when you are using categories acquired from your critical target and rearranging them. Prof. Bruya says that this is “perfectly reasonable to me”. It is not “perfectly reasonable” by the standards of any quantitative-science journal I have ever read. (I welcome counter-examples.) In the absence of such, I continue to think that Bruya’s coyness about this methodology (and his coyness about his methodology on evaluators’ institutions, which he has not yet addressed) is inappropriate as a matter of scholarship.

2) Prof. Bruya resorts to the very worrying methodology of saying that my criticisms are only valid insofar as they also apply to the PGR. (He goes as far as to italicise the points where he feels a double standard is applied.) *This is irrelevant.* Either the reasoning in his paper is valid, or it isn’t. You can’t defend shoddy science by pointing out that others’ science is also shoddy. It’s fine to impugn my motives and object that I only spend time on criticisms of the PGR. (The truth is that the PGR’s defenders don’t make clearly-false mathematical claims, but we can dispute that.) But the question of what I bother to spend my time criticising has nothing whatever to do with whether a given criticism is valid. Does Prof. Bruya think his arguments are valid, or not? If so, let him defend them. If not, it’s irrelevant whether other arguments are also invalid.

3) Prof. Bruya notes that I have not addressed all of the objections he raises. His criticism predates the extension of my criticism to cover his conflation of two categories of institutional association (addressed only in a graph x-axis and never in the text). I didn’t engage with the main additional point he raises (regression analysis of PGR-vs-all-subject-aggregate mark) because I lack the data. If Prof. Bruya is prepared to email me his data-set ([email protected]) I’d be very happy to play with it, with the proviso that I have a baby turning up in the next few days!

...

On reflection I think I spoke too soon when I said that I couldn’t engage with Prof. Bruya’s regression analysis without the data. It also appears to have a severe methodological flaw, though I don’t think this is as serious a point as others I’ve raised.

Prof. Bruya calculates the difference, for each US institution, between (i) the PGR rank and (ii) the “Bruya rank”, i.e. the equally-weighted sum of all speciality marks. He then runs a linear regression analysis to determine the dependence of this difference on the equally-weighted sums of speciality ranking in four areas: History, Value, “M&E” (being Prof. Bruya’s combination of the PGR categories of M&E and Science) and Other.

The problem is that these are very strongly correlated variables: generally speaking, departments with high PGR ranks in one category have high PGR ranks in others, and running regression analyses on very strongly correlated variables has to be done with great care.

To illustrate: suppose I’m interested in whether certain things about US voters (let’s say, their salary) can be predicted by their voting patterns. If one of my regression variables is “usually votes Republican in Senate elections” and another is “usually votes Republican in House elections”, then it’s very difficult to determine the relative contributions of those two variables to a voter’s salary, since each of them is a very good predictor of the other.

You can work around this with a sufficiently large dataset, and there are statistical tests that can be done to ascertain whether a result is still significant even given a high correlation, but Prof. Bruya does not mention any such test.

Brian Bruya fiasco (i.e., "Bruyahaha") update #1

Philosopher David Wallace (Oxford) has officially joined the call for Metaphilosophy to retract Bruya's incompetent hatchet job on the PGR:  https://dl.dropboxusercontent.com/u/8561203/bruya%20letter%20to%20editor.docx.  (Prof. Wallace correctly confines his attention to the methodological flaws, but, as I've noted before (see the addendum), the defamatory content will also require retraction and public apology.)

Meanwhile, a young philosopher in Canada writes:

Very minor comment/suggestion on the recent dustup over the PGR methodology. You’ve referred to it as “The Brian Bruya/Metaphilosophy Fiasco,” but wouldn’t a better name be “The Bruyahaha”? After all, one might aptly describe the entire affair - the article, his sanctimonious response to you - as something of a “brouhaha.” And, in this context, “haha” nicely reflects the appropriate reaction to everything he’s written on the topic.

If you like this, feel free to use it in perpetuity, though leave me anonymous, please - given how high-strung many philosophers on the internet are, it’s not worth it to me to poke this sort of fun in public.

ADDENDUM:  The writer Robert McGarvey, a longtime observer of the philosophy profession, writes:

I assume from your description this is an untenured philosopher in a Canadian school. It is sad that he/she won't put his name on a nice bit of wit. Probably it is also wise and that is sadder.  

 

What has happened to your profession when bland politeness has become the default survival position?​

Brian Bruya's attack on the PGR is even a bigger fraud than it first appeared to be

Philosopher David Wallace (Oxford) has expanded his analysis; I will just excerpt one of the new points he makes:

Without comment, [Bruya] excludes from the survey all features of the PGR concerning the world outside the USA. Since 87 of the PGR’s 303 evaluators had current affiliations at non-US institutions, this discards a non-trivial amount of data.

More significantly, Bruya speaks of evaluators being "from" a given school, or "hailing from" a school, or of "the school with" a given number of evaluators. At least to this reader, the clear impression is that what is meant is the current affiliation of the evaluator. In fact, apparently what is meant is that an evaluator is associated with a particular school either if they are currently affiliated at that school, or if their PhD was granted by that school. This fact is not stated anywhere in the main text, so far as I can see, but I was able to deduce it from the x-axis label in Figure 1 (p.663). It’s problematic that this isn’t clear and explicit in the text, and that vague terms like "from" are used in lieu.

In fact, the correlation between number of evaluators and PGR score is almost entirely due to evaluators’ PhD-granting institutions, and has almost nothing to do with their current affiliation.

Prof. Wallace then re-does the analysis showing how this additional fabrication by Bruya distorts his analysis.

Bruya doesn't speak for scholars of Chinese philosophy

This should really go without saying, but no one should assume that Brian Bruya's incompetent hit piece on the PGR represents the views of all those (or even any of those) who work on Chinese philosophy (a field that I personally wish were better represented, since I have always found work in the area I've read by P.J. Ivanhoe, Bryan van Norden, and Stephen Angle, among many others, iilluminating).  In any case, a graduate student writes to helpfully make the point:

I study Chinese philosophy at the PhD level and I'll be honest: I'm a bit embarrassed by the Brian Bruya stir-up. As much as I would like to see Chinese philosophy more well-received and popular, I can't help but think his methods in this article are manifestly wrong. The rounds of Chinese philosophy grumbling after the publication of each report are nothing new, but that stuff should really stay in "comments" sections of websites, not take up space in respectable journals.

I'm a bit surprised that I'm claiming that Prof Bruya's critique is so misplaced, since he also published an article over the summer in the Chinese philosophy journal Dao that was diagnostic of the field as a whole and its reluctance to take Chinese philosophy seriously---something I thought was common knowledge among Chinese philosophy folk. That article didn't strike me quite as methodologically suspicious (I didn't read it very carefully though), but then again nor was it as polemical or manifestly wrong. 

Blaming the messenger and attacking the PGR I think does more harm then good with respect making Chinese philosophy more respectable, and since my livelihood depends upon that being the case, I felt the need to speak up. That Chinese philosophy is not "higher up on the list" is not the fault of the PGR, but the fault of the discipline as a whole and I would also say, to a certain extent, Chinese philosophy folk themselves. Of course it's an uphill battle to get programs to take world philosophy more seriously, so give them a reason to! As a person in the Chinese philosophy subfield, I can't help but think that Bruya's efforts would have been better spent giving mainstream philosophy a reason to take Chinese philosophy seriously rather than complaining that the discipline's assessors do not.

I would just like to make it clear that at least one Chinese philosophy representative finds the PGR helpful and diagnostic and condemns Bruya's confused critique.

David Wallace (Oxford) on the Bruya critique of the PGR

The philosopher of physics David Wallace (currently at Oxford, moving next year to Southern California) has written up an analysis confirming, with much more precision and technical skill than I could have, my worries about the fabrication of the M&E category Bruya uses, and raising other interesting points.  He concludes:

A large fraction of the “data-driven” part of Bruya’s paper is open to severe criticism on methodological grounds, quite apart from one’s assessment of the more qualitative issues. The severity of the criticisms are such that it’s very hard to see the paper passing peer review in any journal of the quantitative sciences. Indeed, the first and most severe criticism in this note – the silence, in the main part of the paper, about the reclassification of the PGR “Science” category as M&E – would in other contexts be troublingly close to academic malpractice.

I don’t use that term lightly and I don’t intend any accusation of malice to Professor Bruya. But as a discipline Philosophy needs to be extremely concerned if it allows publication of material that uses the methods of other academic disciplines but which fails to pass the basic methodological standards of those disciplines.

Brian Bruya's alleged critique of the PGR

Brian Bruya, who teaches philosophy at Eastern Michigan University and earned his Ph.D. at the University of Hawaii (specializing in Chinese philosophy), has penned a purported critique of the PGR, which an actual journal, Metaphilosophy, published (readers will see in a moment why this is surprising).  Why do I mention these facts about the author?  Because the author has reasons to be hostile to the PGR--he plainly feels he and his friends in the profession are undervalued because of the PGR--it should not be wholly surprising that the critique the author produces involves mistakes, shoddy arguments, and fabrications. 

This is no longer my headache, but since the article ignores the participation of the Advisory Board in producing the report for the last 15 years, and continually personalizes his critique to me, let me note the following:

1.  Bruya asserts (674), falsely, that the "Metaphysics & Epistemology" (M&E) category in the PGR includes 15 sub-specialties, more than "Value Theory" and "History of Philosophy" together.  In fact, the M&E category has only 7 specialties listed, compared to 6 for Value Theory and 9 for History of Philosophy.  In fact, there are 8 specialties under "Philosophy of the Sciences, Mathematics, and Logic," which is the fourth major sub-division of the field the PGR has used for a long time; these four divisions correspond quite well to the areas represented by about 95% of philosophers in the Anglophone world (most of the rest are picked up by the areas that don't fit neatly into any of the others, because they cross divisions, like Feminist Philosophy or Philosophy of Race).   Buried in an appendix at the end, Bruya finally acknowledges conflating the divisions, with the explanation that, "It is uncontroversial that many of the specialties of M&E, philosophy of mathematics, and logic are core specialties of Analytic philosophy" (686).  This is, of course, revealing about Bruya's biases, and his lack of understanding of "analytic" philosophy (he might talk to some philosophers of physics and biology to find out what they think of a lot of, say, contemporary metaphysics).  But the mistake vitiates in its entirety the "analysis" on pages 668-678, where the statistical analysis purporting to show "bias" in the evaluator pool depends entirely on his massive inflation of the M&E category.

2.  Snowball or chain-sampling is a perfectly appropriate method of sampling when what you want is a kind of "insider's" knowledge.  Bruya's argument against it is silly:

The reason [snowball sampling] is used is as an expedient way to access a hidden population, such as social deviants (drug users, pimps, and the like), populations with very rare characteristics (such as people with rare diseases, interests, or associations), or subsets of populations associated in idiosyncratic ways (such as networks of friendships)....Philosophers are neither social deviants nor difficult to find, as every philosophy program's faculty list is public information.  (660-661)

It does occur to Bruya that perhaps what is wanted is expert and well-informed opinions.  Bruya comments:  "the very earning of a Ph.D. is the academic standard for expertise" (661).  If that's one's view of expertise, then the PGR will not make much sense.   Later in the article, however, Bruya acknowledges that,

We want experts to provide their opinions when expertise is required for a sound assessment, and we would not insist on getting a representative sample of all such experts.  We see this all the time in academia.  We have PhD committees, tenure review committees, grant committees, and so on, which are formed for the purpose of providing expert evaluation.  And for none of these do we insist on getting a representative sample.  (667)

Given this admission, one might wonder what all the fuss is about?  Bruya again points to "bias," but as we've seen (#1, above) the central and allegedly data-driven analysis of "bias" in the evaluator pool involves a straight up fabrication about the areas of philosophy represented by the evaluators.  

3.  In an unrelated effort to show "bias," Bruya asserts that there is a category of philosophers who are "Methodological Continentalists" (665-666) which would encompass programs like DePaul, Duquesne, and Emory (666).  SPEP folks have long maintained, of course, that they are insulated from normal standards of philosophical scholarship because there's something putatively distinct about the work they do that makes such standards irrelevant.  Bruya is entitled to endorse that myth.  But what is, again, pure fabrication is to assert that "Leiter refers to this brand of philosophy in his own published work," noting the introduction to The Oxford Handbook of Continental Philosophy I wrote with Michael Rosen (666).  Bruya gives no page reference, because there is none that would show that we recognize something called "Methodological Continentalists" represented by departments like DePaul, Duquesne, and Emory.  How such a naked fabrication got through the peer review process is, again, mysterious.

4.  Bruya repeatedly misrepresents Kieran Healy's research about the PGR; readers interested in Prof. Healy's views can start here.

5.  Bruya's main methodological suggestion (681 ff.) is to aggregate scores in the specialty areas for overall rankings.  The Advisory Board discussed this in past years.  Since there is no way to assign weights to the specialty areas that would not be hugely controversial and indefensible, the PGR has never adopted such an approach.  Bruya has no real solution to the problem (though one may rest assured Chinese Philosophy will count for more!).

I predict, with confidence, that no changes to the PGR methodology are likely to result from this very confused critique. 

I also trust the journal Metaphilosophy will withdraw the article in its entirety given the fabrications, and subject a revised article to a more serious peer-review process, to insure the final version is not so obviously shoddy.

ADDENDUM:  I suppose I should say a word about Bruya's defamatory claim that "one cannot help but wonder whether the PGR's hidden biases are based in sexism, racism, ethnocentrism and xenophobia" (679).  Obviously, Brit Brogaard was motivated to join me as co-editor of the last PGR by the opportunity to help propagate "sexism, racism, ethnocentrism and xenophobia."  So were the women, African-Americans, and Hispanics on the Advisory Board; and the dozens of women, African-Americans, and Hispanics who participated as evaluators.  An alert reader might notice that Bruya motivates this smear first through his fabrication of a huge M&E category (see #1, above) and second through his unmotivated claim that "a program that was both complete and perfectly balanced...would have a total of thirty-three faculty members," one for each of the 33 specialties ranked in the PGR.  Why should any program aspire to represent every specialty in the PGR?  Bruya never explains.  But, of course, if you start with that assumption, then it looks like Chinese philosophy, not to mention philosophy of religion and medieval philosophy and philosophy of art, get shortchanged.  It never occurs to Bruya that this might be a fact about the profession that the PGR records, rather than creates (in fact, during the PGR's lifetime, the range of philosophical areas well-represented in the profession has increased, and that's been reflected in the PGR).

DEC. 15 UPDATE:  After this update, I am not going to spend any more time responding to Brian Bruya, who in 14 single-spaced pages tries to create the appearance that he is responding to the criticisms, above.  In some cases, it is obvious that he has no response, posturing to one side.  I'll just make seven quick points:

1.  Bruya claims that the "reason I wrote the critique of the PGR is that I see a group of very intelligent and apparently well-meaning people involved in an influential publication that  ultimately locks out"  non-Western philosophy.  These are the same people whom Bruya smears in the article as having hidden biases deriving from "sexism, racism, ethnocentrism and xenophobia."

2.  I noted that "the article ignores the participation of the Advisory Board in producing the report for the last 15 years, and continually personalizes his critique to me."  Bruya drops everything after the comma, and then responds, irrelevantly, that his article mentions the Advisory Board in a couple of places.  Anyone who bothers to read Bruya's article can see very clearly that throughout he treats the PGR, and all the bad things about it, as due to me, personally.  He ignored the fact that it has been, for a long time, a collective effort.  Thus, he does, indeed, "personalize[]" the critique.

3.  Bruya's M&E category is a fabrication, and as I noted, he does finally acknowledge it to be so in an appendix at the end of the paper.  The fabricated category allows him to claim that there are more M&E categories than value and history categories.  It also vitiates his statistical claims, which trade on the huge M&E category he created.  There's no doubt that, as Kieran Healy has shown, that metaphysics, epistemology, philosophy of language, and philosophy of mind are high prestige areas in Anglophone philosophy; the PGR records that fact, it didn't create it (a point on which Bruya continues to be silent).  That does not justify Bruya's fabrication, which he still doesn't defend or even explain.  (I did laugh out loud when the guy who smeared me and the PGR participants as infected by "sexism, racism" etc. worries that it might be ad hominem to note that he doesn't seem to understand what "Analytic philosophy" is.)

4.  I noted that the 33 PGR specialties represent 95% of what philosophers do.  Bruya repeats the non-sequitur that an APA survey (of only participants in one Eastern Divison meeting, the Eastern Division being the most atypical of the three Divisions) would have resulted in a different distribution of specialties.  Many (most?  I haven't checked them all) of the specialties that Bruya labels "other" are, in fact, part of existing specialty categories.  If my 95% estimate is wrong, we still haven't seen the evidence.

5.  Bruya manages never to respond to the points, above, about expertise, though assures us that he thinks his argument was a good one.  Readers can decide for themselves.

6.  Bruya repeats the falsehood that Leiter & Rosen endorse his version of Methodological Continentalists, as exemplified by programs like Emory, Oregon, and Duquesne.   He now produces a quote which shows nothing of the kind.

7.  Bruya thinks an alternative way of producing an overall ranking is to simply "sum" all the specialty scores.  I confess it never occurred to me or anyone on the Advisory Board that this could possibly be a sensible way to proceed.  Readers may decide for themselves whether Bruya is persuasive on this point.

Bruya, I suppose, is counting on the fact that the appearance of a response, complete with school marmish posturing, will suffice, and that readers won't examine his confused paper carefully enough to see what's really going on.  Maybe he is right.  Nonetheless, I continue to predict with confidence that Brit Brogaard and a new Advisory Board will not adopt any of the suggestions offered by Bruya because she is smart and will appoint a smart Advisory Board, and it's easy enough for any smart person without an axe to grind to see that the criticisms are mostly confused or tendentious (not all, some are familiar, but not significant).  I am also hopeful that Metaphilosophy, which allegedly has a peer review process, will withdraw the article in light of the fabrications and defamatory smears.

ANOTHER:  Unfortunately for Prof. Bruya, the philosopher of physics David Wallace did examine the paper carefully, and his conclusions are damning.