Time to End In-House Editing of Leading Philosophy Journals (Leiter)

MOVING TO FRONT FROM AUG. 28 TO ENCOURAGE MORE DISCUSSION (and do read the existing comments, which are of high quality and quite interesting)

Patricia Smith Churchland at the University of California at San Diego and Daniel Dennett at Tufts University write with the following apt comments about the editorial practices of some of our profession's most visible journals:

Several of us old guys have often discussed the unprofessional nature of the so-called premier journals in our discipline. We shall restrict ourselves to mentioning Journal of Philosophy and Philosophical Review. First, both are "in house" journals, meaning, as we understand it, that the editor (at Columbia and Cornell respectively) standardly selects faculty from his/her department to review papers. Papers are not standardly sent out for peer review. This makes the journals vulnerable to a certain kind of corruption and cronyism that would not be tolerated in the sciences. We know of no first ranked journal in the natural or social sciences that operates that way, or second-ranked journal either, for that matter. Our casual survey of scientists revealed that the "in house" refereeing system is regarded as completely unprofessional. Obviously the "in house" policy also means that from time to time people who are not particularly competent are reviewing submitted manuscripts. The policy may also partially explain the absurdly long time it takes for manuscripts to be reviewed.

May we also add that the journal  -- Philosophical Psychology -- whose editor, Bill Bechtel,  is in the department at UCSD,  does not operate as an "in house" journal but according to the professional criteria in the natural and social sciences. Philosophy of Science also abides by a peer review practice, and its location moves as a function of its term-limited editor, who is selected by the Philosophy of Science Society.

The "in house" practice of refereeing at Journal of Philosophy and Philosophical Review is likely an innocent relic of earlier times. It is easily corrected.

I recall a time in the early 1990s (around when I cancelled my subscription) when we used to refer to the Journal of Philosophy as the Journal of Philosophy and Decision Theory, given the preposterously large number of papers on decision theory it published, due to the influence of Isaac Levi, then a senior member of the Columbia department and an influential editor of the Journal.  Another pernicious, it seems to me, aspect of the in-house editing at J.Phil. in particular is that its rather precious book review space is given over disproportionately to books by Columbia faculty.

I wonder, though, what other philosophers think?  I am inclined to think that Professors Churchland and Dennett are right, and that it would be in the interest of the profession, and the editorial integrity of these journals, for a radical revamping of their editorial practices:  the Ethics model created by Brian Barry, and continued to the present, seems a better one.  What do readers think?  As with our earlier, informative thread on irresponsible journal practices, postings will have to be non-anonymous.

UPDATE:  Do see the comments by Nicholas Sturgeon and Brian Weatherson, the current editors of Phil Review, regarding that journal's editorial practices.

Comments

[Colin Farrelly]

Brian,

I think this is an excellent topic that the discipline needs to seriously consider. I mentioned my reservations about the practice of "in-house" journals in a comment to a previous post by Jason (on tenure and academic evaluation) back in November

http://leiterreports.typepad.com/blog/2005/11/tenure_and_acad.html

Here is the gist of my comments then, which overlap with those expressed here:

"One obvious difference between political science and philosophy journals is that a number of the most prestigious “general” philosophy journals are edited “in house”, with the editors and reviewers (either some or possibly all) consisting of faculty from one particular department. Compare, for example, the highest impact Political Science journal- American Political Science Review- with the Journal of Philosophy or Phil Review. The American Political Science Review is the highest impact journal and considered the “flagship” journal of the discipline. It publishes articles in all areas of the discipline and has a rigorous external (double—sometimes triple) blind review process. This contrasts in important ways with flagship journals in philosophy that have an “in-house” vetting process (which they utilise either exclusively or as a major component of the evaluation process). I think this is troubling for a number of reasons. It might disadvantage people who do certain areas of philosophy that no faculty member at the institution in question does (or if they do, they might not be sympathetic to the position/methodology you advocate). Relying on the practice of in-house reviewing from one particular dept. is no doubt going to have an unjustified limiting impact on the papers accepted for publication. This has perverse consequences— in effect it makes it the case that a necessary condition of publishing a paper in one of these two flagship philosophy journals is that someone at one of these two institutions likes your work. I don’t think this ought to be the case. The flagship journals of a discipline should rely on blind external assessors, much like the reviewing process of Ethics. If the Journal of Philosophy and Phil Review could shift their editorial practice to be more in-line with Ethics then I think the process of publishing in the flagship “general” philosophy journals would be more inclusive and representative of the discipline".

Cheers,
Colin

[Aldo Antonelli]

I wholeheartedly agree with the above. Let me bring to bear the example of one of the premier journal in my specific subdiscipline, the Journal of Philosophical Logic. JPL is published under the auspices of the Association for Symbolic Logic, which appoints a board of 5 editors to 3-year terms. One of the editors is selected as coordinating editor (primus inter pares, as it were). Let me point out that the 5 JPL editors are purposely scattered at the 4 corners of the world. Submission are received centrally but then the process is the complete responsibility of the editor who elects to handle the paper. The editor is in charge of selecting an external referee and making the final editorial decision. Accepted papers are then sent back to the coordinating editor to be sent on to the publisher (in this case, Springer). The system is far from perfect, and we are not immune to long refereeing times, but I believe at least any bias is mostly avoided and certainly more accidental than built-in. Let me finally point out that the board answers to the ASL Council that can decide not to renew an editor's term (or in extreme cases to remove the editor altogether).

[Robert Talisse]

I recently had an absolutely *awful* experience at JP with a paper that critically engaged the work of one of the Columbia faculty. They held the paper in review for well over a year (15 months), and eventually flatly rejected it without any official comments: I was sent parts of an email exchange about the paper between two or more unidentified persons, with a note from the general editor kindly apologizing for the fact that the exchange would be of little help to me in revising the paper (btw, he was correct in this). The general editor also told me-- after I had contacted him in the 8th month that the paper was under review-- that part of the hold up was the fact that the paper concerned one of their own. I'm of course in the dark about the precise nature of this conflict, and I have no idea why their process apparently broke down in the case of my paper (they publish lots of stuff concerning their own people), but I can't help but think that this would not have happened had JP not been reviewed in house.

--Robert Talisse

[Margaret Atherton]

I see this undoubtedly serious problem as a problem that can be traced back to the APA. In both of the examples of superior journal behavior listed above, the journal in question has an institutional relationship with the appropriate professional organization. If the premier professional organization of the philosophy profession took it upon itself as one of its responsibilities to sponsor one or more premier journals, then we might not be in the anomalous situation in which one or more house journals have apparently in perpetuity laid claim to the position of "best journals in the field."

[Brit Brogaard]

Personally I would prefer if all journals implemented a tripple blind refereeing policy. The idea would be that the submitted paper is assigned a number by the editorial secretary before the paper is handed over to the editor. The author's identity will then, at least in ideal circumstances, be unknown, even to the editor. It is my understanding that *Mind* is already refereeing papers in accordance with this general policy.

Having said that, I should also say that I am somewhat sympathetic to the in-house journals' refereeing policies. Ideally, no one would know the identity of the author until a decision had been reached. But things are not ideal. I often know the identity of the authors whose papers I am refereeing. So, in the end, I don't think there is a significant difference between the refereeing policies of in-house journals and other journals.

[Andrew Mills]

Two comments:

(a) If the philosophical community begins to believe that the method by which these journals referee their submissions is flawed, it would seem to me that the philosophical community can begin to lessen the esteem in which they hold these journals. It is not written in stone that JPhil and PhilRev are the top journals. We, as a community, are the determiners of their status, are we not? Of course, if the reviewing situation is flawed, then the community should lobby the editors of those journals to change their practice, but it also seems legitimate to be boosters of journals with better editorial practices. Of course, this is not the sort of thing that be changed overnight, but, if the Gourmet Report has taught us anything, it is that our community is in charge of the reputations of its institutions. Just as we no longer reflexively look to Harvard as the best philosophy department in the country, perhaps we should no longer look to JPhil and PhilRev as the top journals.

(b) One of the arguments for the flaw in the current JPhil and PhilRev editorial practice is that specialized submissions might be inadequately refereed (if no one with the relevant specialization works at the relevant dept.) It is interesting that journals from the sciences and social sciences were mentioned as models that philosophy journals should emulate; I wouldn't want the assumption that philosophy should look to the sciences as its model to go unquestioned. Should we be promoting ever more specialization in our work?

[Seiriol Morgan]

I don't know what happens at the JPhil, but with respect to the Phil Review at least the account quoted above from the Churchland/Dennett letter is incorrect. What happens, according to their website at any rate, is that there is an initial review in which each submission is read by two members of the Cornell department. Those that pass this first stage are then sent out to an "expert" referee. So assuming they are serious about the "expert" bit, nothing gets published without it being scrutinised by a top specialist in the field. I imagine that since they are the Phil Review they are serious; you wouldn't expect them to have trouble finding good people to referee for them. Details here:-

http://www.arts.cornell.edu/philrev/info/editorial.html

Of course, one might well think that this policy is itself vulnerable to criticism. But it is not the bastion of insularity that has been suggested.

With regard to Brit Brogaard's worry about blind review, PR operates exactly the policy she favours. (Though it didn't seem to me that this was what Dennett and Churchland are exercised about. Their point seemed rather to be that the wrong people are doing the refereeing.)

[Colin Farrelly]

But Brit the main problem with in-house edited journals is not so much the lack of blind review (indeed some in-house edited journals have a policy of blind review).

Rather it is the "limited pool" of referees/assessors. I believe that is the most significant difference between in-house journals and those that rely (primarily) on external blind reviews.

A necessary condition (perhaps sufficient?) of getting a paper published in JP or Phil Review is that a faculty member at Columbia or Cornell likes your paper. That is troubling for a whole host of reasons.

Cheers,
Colin

[Keith DeRose]

I have some comments, mostly in defense of in-house journals, but they got too long to put them here. So I put them here:
http://bengal-ng.missouri.edu/~kvanvigj/certain_doubts/?p=613

[Steve Hales]

I agree with Churchland and Dennett. In-house reviewing is prone to insularity, inefficiency, and occasional incompetence. Also troubling, but not yet commented on, is the extreme collection of editorial power and influence wielded by some departments. PPR and Nous are moving to Rutgers, which already has Oxford Studies in Metaphysics and pseudo-control (since John Hawthorne is part-time now at Rutgers) of Oxford Studies in Epistemology. That's a lot of influence over what approaches are deemed worthy and which positions taken are deemed timely.

[Eric Schwitzgebel]

It would be a shame to mess up a process that's working to produce excellent journal issues, year after year. So I think part of the question has to be about the quality of the final product at J Phil and Phil Review.

[To m Hurka]

Keith's post is characteristically thoughtful, and we should acknowledge that there are distinctive advantages even in a single-editor journal, like Mind under G. E. Moore or, more recently, Simon Blackburn. But on balance I'm inclined to agree with Churchland and Dennett.

The various posts make at least three charges against in-house journals:

1) they're more prone to cronyism and corruption,

2) they're more likely to use referees who are incompetent either a) generally or b) on a paper's specific topic, and

3) they're more likely to favour papers on a narrow range of topics (see e.g. Brian's comment about the JP and decision theory).

Re 1), Keith says there can also be cronyism in non-house journals, given how they select referees, and that's true. But I think the danger is less in those journals, because the power to be cronyish is distributed among more people, i.e. all the referees rather than just the folks in one department.

Re 2a), about the general competence of referees, Keith is probably right that in-house journals have an advantage, at least if they're in a top department. When I was an editor at the Canadian Journal of Philosophy we would use something like 250 different referees over a three-year period, and I don't think I could claim that their average philosophical ability was equal to that of, say, the Cornell department. But in-house journals are at a significant disadvantage re 2b), since no department will have specific expertise in all the topics papers are submitted about. A non-house journal is much better placed to have its submissions read by people who are knowledgeable about their specific topics. And that's surely important.

Re 3), which Keith doesn't mention, in-house journals are again more likely to favour a narrow range of topics. Unless one is a saint (like G. E. Moore?), one's judgement about a paper is bound to be affected by one's views about what's an interesting topic, and an in-house journal gives that influence to a smaller number of people's views. In this connection Keith's citing of Philosophy and Public Affairs is illuminating, since despite its general excellence that journal has had an unmissable tendency to publish a disproportionate number of papers that either use or discuss the views of Rawls.

[Thom Brooks]

A few comments as a fan of both J.Phil. and Phil.Rev.: for one thing, I think both journals have managed to publish some terrific material. (I'm currently enjoying a PR article on crime.) The worry doesn't seem to be that either journal is publishing poor material: we are all their readers and clearly interested in their contents (and contributing to them). The worry seems more focussed on the range of material covered. (Although I'd never justify any journal taking so long to referee pieces---Talisse's story is all too common and genuinely horrible.)

Unless a faculty covered every possible area of philosophy (which perhaps only the smallest handful do), then to judge all competent work in philosophy with sufficient expertise would most likely require going to external faculty for their advice on said pieces. I agree with previous posts that DeRose may be quite right to point out some cronyism amongst editors in terms of the kinds of referees (or perhaps small circle of referees) they ask to help out, but wrong to think this problem worse with non-in house journals. So long as the pool of referees are diverse (after all, there are a great number of excellent philosophers to choose from), then worries over cronyism disappear.

On a different note, many have noted before how these said journals publish work on narrow topics. Perhaps different faculties were influenced by particular members of staff who left a clear mark on the kinds of work published. However, it is also sometimes the case that a journal is bound by the material it receives. One would imagine PPA, for example, quite hard pressed to not find top work in the field that didn't make some mention of Rawls---heck, it's still difficult (and I do it too). Rawls is a major figure worth the attention. Perhaps a journal's board takes the decision to publish work with a certain slant. However, perhaps the journal is doing nothing suspicious at all---the work it publishes simply represents the best work sent to it, much of it taking issue with a particular slant that is of interest to the majority of persons working in the area more generally. It's hard to say.

Finally, I'd be all for the APA taking a flagship journal...or two, preferably from amongst existing journals. I think this would be terrific.

[Mike DePaul]

Just a comment about the competence of referees. Let's go along with Tom Hurka's claim that the average philosophical competence of the members of a department such as Cornell or Columbia is higher than that of the referees used by other journals. I can say from experience that it does not follow that the quality of the refereeing will be better. For one thing, evaluating philosophy papers is not exactly the same thing is producing them, and the former skill is required in a referee while the latter is what goes to making a department with high average philosophical ability. But perhaps more importantly, the quality of the work a referee does is to a significant extent determined by the amount of time and effort the referee is willing to put into the job. Over the two years I have been editing the Journal of Philosophical Research, admittedly not a premier journal, I have been struck by the lack of correlation between the quality of a referee's own work, or reputation, and the quality of the reports. Some very highly regarded senior philosophers promptly produce splendid reports with detailed, helpful comments for authors, but others do not. But just as often solid journeyman philosophers produce equally splendid reports. Combining this with the fact that the solid journeymen tend to agree to referee articles for me more often, I find myself seeking them out to serve as referees rather than very highly regarded senior philosophers, Indeed, I tend to write to the latter mostly for help in identifying the former.

[John McCumber]

Churchland and Dennett are right, as are the comments in this space questioning the role of non-peer reviewed journals in philosophy. (We can add the "Review of Metaphysics" to the list).

I wrote about this in an article in The Journal of Practical Philosophy for 2000. One point there which should be made here concerns the way in which review by editors, rather than anonymous experts in the field, increases the control of the elderly over the profession.

I have edited a philosophy book series for many years; like all book series, it is subject to peer review. I can testify, then, that there are two kinds of covering letters accompanying submitted manuscripts: those which say "So-and-so advised me to send this to you," and those that do not. "So and-so" is very often the author's thesis advisor, and is usually someone for whom I have friendship and admiration. Usually too, they have a certain degree of power within the profession.

But none of that matters, because the ms goes out to an anonymous expert--not someone from a small group, such as a department, but truly anonymous--and I have to abide by their judgment. If they pan it, I can't publish it. If they praise it to the skies, I have to accept it.

I don't know, but I suspect it would be rather difficult to tell a powerful and influential friend of mine that I had rejected the ms of some student of theirs. I'm sure that happens all the time--the point is that the structure of non-refereed journals is weighed against it. How many people in philosophy owe their careers to articles in non-peer-reviewed journals? How many of those articles were advanced at those journals because of the recommendation of a senior colleague of the Editors? And how many good articles did not get published, how many careers didn't happen, because of the lack of such recommendations?

Philosophy is the only field in the sciences and humanities which operate this way. Law journals, to be sure, are often not peer reviewed. But they are edited by law students, so the young are judging the old. In philosophy it is the reverse.

I think this is a very serious issue--it has repercussions throughout the profession (or "profession," for until the situation changes I do not class philosophy as a true academic discipline at all). Thanks to everyone for bringing it up and thinking about it.

[Nicholas Sturgeon and Brian Weatherson]

As the current editors of the Philosophical Review, we thought that we might comment on the recent message from Professors Dennett and Churchland, to clarify a few points about how the journal operates. It’s quite true that it’s edited (as we say on the masthead) “by the Faculty of the Sage School of Philosophy at Cornell University.” What this means in practice is that the two editors—there is never just one—come from among the tenured members of the department. It’s also true that the editors frequently call upon their colleagues, when they have the appropriate expertise, to referee papers. Still, they have for as long as we can remember also sent very many papers to outside referees, and for some time they have been sending quite a few more papers to outside referees than to readers in the department. The aim is always to tap appropriate expertise, whether inside or outside our department; they don’t turn papers over to “people who are not particularly competent”. (In any case, we notice, Dennett and Churchland use the term “peer review” in a puzzling way, as if it didn’t apply to cases of in-department refereeing. Why does it count as peer review if we send a promising paper in the philosophy of mind to Ned Block to read, but not if we give it to Sydney Shoemaker? Or if we send a paper on Aristotle to John Cooper but not if we give it to Gail Fine or Terry Irwin? We are a good enough department to have a number of recognized experts on hand, and we see nothing “unprofessional” about having them referee a few papers—it’s never more than a few—every year.)

We also wonder whether the Review differs very much in the use of departmental colleagues from the practice of other journals that aren’t legally tied to a single department. One of us (Weatherson) was recently a member of the Brown department, where he was regularly asked to referee as many papers for each of Noûs and PPR as any member of our department (including him, before he began a term as editor) is asked to read annually for the Review.

We are puzzled, too, by the suggestion, perhaps meant merely as an afterthought, that it may be in-house reviewing that accounts for long delays in the handling of manuscripts. In our experience precisely the opposite is true—it’s external referees who account for most of the long delays. (As we have said, this doesn’t keep the editors from using more external than internal referees, but it’s a recognized headache.) You can see why. If we give a paper to a colleague, that person has to meet our gaze in front of the department mailboxes several times a week, and reports are usually done promptly. If an external referee is dilatory we send emails, but emails can be ignored.

It is hard to reply to unspecified suspicions of “corruption and cronyism”. But note the decades-old policy, stated on the masthead and strictly adhered to, that no paper by a Cornell faculty member may be published in the Review except on recommendation of an outside referee. And since both editors and referees read all initial submissions with the author’s name removed, it would be hard to favor our friends at that point, either in editorial judgments or in the choice of referees, even if we wanted to. (And, again, we don’t see how the potential for this sort of abuse, if someone is really determined to engage in it, is any greater at the Review than at many other journals not tied to a single department.)

One of us (Sturgeon) is old enough to have heard tales from the 1930’s, when editorial decisions about the Review were made at department meetings at which submitted manuscripts were passed around for the faculty’s perusal. That was a bad practice and the journal at that time was a mediocre one. One of the good things that Norman Malcolm and Max Black did for the Cornell department, when they arrived after WWII, was to turn the Review into a thoroughly professional journal, with a careful, disciplined editorial routine that included a lot of reliance on external as well as internal referees. We’re pleased to be continuing that editorial tradition.

[Chris Brooke]

I'd be careful before praising the American Political Science Review as a model of how to produce a leading journal. While it may be sponsored by the APSA, blind-refereed and so on, the unpopularity of the journal among many political scientists and the widespread feeling that certain methods and approaches (quantitative, formal) were being wildly privileged over others (qualitative, especially) was a major reason as to why the Perestroika movement caused such upheavals in the APSA a few years ago.

[Tad Brennan]

Is this discussion confusing the roles of referees and editors?

The job of the referee (wherever located) is to decide whether a paper is publishable.

The job of the editor is to decide what gets published.

It's the editor (or editorial board) that makes the final determination of the contents of each volume.

I imagine that in most cases, editors have the room only to publish N papers, but receive 2N papers that are judged publishable by referees (or maybe its a different factor--or an additive term! or an exponent! help yourself!).

So long as the referees say that more papers are publishable than can be published, the editors are going to need to exercise discretion over what, in fact, gets into each volume.

That means that some of the "favoring of topics", the "narrow interests", and so on, are really determined by the editorial decisions.

Not by referees decisions, and so, a fortiori, not by whether the referees are in-house or out.

I suppose that an opponent of in-house editing might say that the favoring and narrowing can be intensified by in-house refereeing, if the preferences of the editors are amplified by the preferences of their colleagues who referee and who share their interests.

But much of the effect of editorial preference will still operate with external referees. If I send my referees 200 papers and they tell me 40 of them are publishable, I still have to winnow that down to the 5 to 10 that will appear on the table of contents.

So: I don't think the fact that certain journals favor certain topics is really one that has much to do with whether refereeing happens in-house or out. It's a function of editorial discretion, which is exercised after the referees have had their say.

(I should say that I myself have never been a lead editor on a journal (I have done too much refereeing, I have been on editorial boards, and I am an associate editor now). If lead editors want to tell me that they make up the final table of contents in a very different way that rules out the effect I have described, then I will defer to their knowledge.)

[Colin Farrelly]

Chris’s reference to the Perestroika movement in the American Political Science Association is helpful because the comments from Dennett and Churchland perhaps suggest that a similar movement in philosophy is needed (and has started?).

Those unfamiliar with what occurred in Political Science can read about it here:

http://chronicle.com/free/v48/i04/04a01601.htm

Like the perestroika movement in the American Political Science Association, many philosophers will resist changing the status quo. But I think those taking this position face a formidable task. Given that no other academic discipline has first ranked journals that are edited in house, the burden of proof is on those who defend the status quo.

Why is philosophy special? Why should we believe that academic philosophers (even those in excellent departments) are not susceptible to insularity, corruption and cronyism?

Furthermore, what do we have to lose by reforming current practices in the first rate journals? It seems like a “win/win” situation to me. No doubt the two editors of Phil Review must find it very onerous to read the 300 or so papers (in all different areas of philosophy) they receive each year. That is almost one paper a day! I assume they still have teaching and other duties to do in addition to reviewing all these submissions on all areas of philosophy. Why not make life easier on themselves by creating an editorial board consisting of academics from different universities with expertise in all the areas of philosophy? What is the reason for resisting this change?

So far the only answer seems to be that external referees take too long to referee a paper. This is a problem for all academic journals (it's not unique to philosophy, let alone Phil Review). If the journal’s external referees take too long to review a paper the journal needs to find better referees.

I have not been persuaded that the status quo has any virtues. On the contrary, the comments raised here suggest that it has many vices.

Cheers,
Colin

[Steve Hales]

It seems to me that the place of "journals" that are almost exclusively invitation-only is worth debate in this forum, especially Midwest Studies, Philosophical Perspectives, Oxford Studies in Metaphysics, Oxford Studies in Epistemology, Philosophical Topics, and Philosophical Issues. There may be peer (even blind, external) review of the articles in these venues, but seriously, how many invited pieces get rejected?

[Thom Brooks]

Two quick points to add:

(1) I agree with Tad Brennan on much that he has to say, with one small exception. It is the case that each journal issue has space for only so many articles. At least in the case of the Journal of Moral Philosophy ( http://mpj.sagepub.com ), we publish all articles that referees recommend. My suspicion is that the majority of journals work like this, too. Thus, if referees claim X articles are publishable, they're put in a queue until we can squeeze them all into print.

(2) The second point I wanted to raise was to thank Philosophical Review's editors for their clairifications. As I suspected, the journal is not entirely refereed in house and I think the points raised by the editors quite satisfactory. I honestly don't see what particular problem philosophers might find. As I noted previously, this debate is curiously *not* about the journal's contents which we all surely agree are top notch. If the contents are of such a high standard, why fix something that ain't broke...?

[Keith DeRose]

I thought our topic was the advisability of having the top journals in our profession being run "in-house," the way the Phil Review and the JP are. We now have been given a good deal of information by the editors of one of those journals how it is run. Since I don't know enough about how the JP does things, I'll focus on the Phil Review, about which we now all have a pretty good idea.

And it turns out that, as far as the reviewing of papers goes, it isn't run all that differently from other journals. Not only do they use referees outside of the journal's host department (Cornell), but they use them a lot. On this score, I may have over-stated the extent to which they operate in-house in my Certain Doubts post (linked to above). This should take care of the worry about using referees ill-suited to the task. Tom Hurka (above) divided this issue into two parts: "2) they're more likely to use referees who are incompetent either a) generally or b) on a paper's specific topic." He concluded that while in-house journals may fare better than other journals on 2a, they're likely to do worse on 2b, given that no department will have experts on all the topics that they receive papers in. But it now looks like a journal like the Phil Review may have the best of both worlds here. Supposing the hosting department has excellent philosophers in several areas (which Cornell does), and supposing that potential excellent refs are more likely to accept an assignment for a job for a journal hosted by their own department, the journal gets an inside track on some excellent refs. And if they go to outside refs whenever they don't have an excellent in-house candidate, where is the disadvantage? (Indeed, since, as the PR editors point out, editors of other journals can and do also use refs from their own departments, it's not even easy to pin-point any difference here.)

The real worries seem to concern having an editorial board all coming from one department, as opposed to one comprised of philosophers from several departments. There may be room for worries about narrowness here, but I wonder. Departments can, and arguably Cornell's does, have a wide variety of areas and approaches represented in them. Will they tend to be narrower in some bad way from other journals? It's hard to even know how to begin to answer that without specifying how the editorial boards of the other journals are selected. Of course, it's entirely possible for an editorial board consisting of philosophers from many departments to be much worse in terms of bad kinds of narrowness.

At any rate, Steve Hales put his finger on what seems to me to be the real pressing publishing issue for our profession:

"It seems to me that the place of 'journals' that are almost exclusively invitation-only is worth debate in this forum."

This was discussed in a long comment thread at Brian Weatherson's blog here:
http://tar.weatherson.org/2004/12/07/philosophical-perspectives/
See, for example, my comments there, numbered 28, 44, and 73. What seems important to me (in addition to giving more people a fair shot at invitations to the invited venues) is that very many of what are our discipline's most prestigious publishing venues operate by blind review, where someone without much status in the prfession can send a paper that will be reviewed by editors & refs who are unaware of her (lack of) status, and get a fair shot at publication. And what we can do to further that is to, in various ways, support journals that operate in that "blind" way. So, in what seems to me the most important issue, the PR and JP are on the good side, and, along with many other journals, deserve our support.

[Brian Leiter]

I'm a little surprised by Keith's Panglossian read on the in-house situation, though I agree that Phil. Review is less in-house than the original letter from Professors Churchland and Dennett had supposed, as the helpful remarks of Professors Sturgeon and Weatherson makes clear.

It is still the case though, as far as I can see, that Cornell faculty referee a lot of the papers, and Cornell faculty choose the referees, which gives the Cornell faculty a rather remarkable degree of influence over what appears in one of the most visible journals in the profession. That also means it is a very different operation than, say, Ethics. And the worry is less about competence of referees, than reinforcing the view of folks at Cornell (or Columbia, in the case of J.Phil.) about who is competent.

Phil Review publishes very good work, more consistently than, say, a fairly corrupt journal like Phil & Public Affairs, where folks in the "inner circle" can get their third-rate work printed (though even PPA seems to be reforming in recent years). The issue is what very good work isn't Phil Review publishing because of the in-house editorial practices? I'm not sure about the answer, but I would think that's one part of the question that Churchland and Dennett were raising.

[Lee Walters]

It seems that Keith has summed this up nicely, at least from my perspective as a pre-publication graduate student. The main sources of concern for aspiring philosophers I would guess are invitation-only publications and lack of blind review at journals, both of which may be perceived as working against new philosophers.

It would seem that we have an opportunity to have an influence on the latter by collating and publicising the reviewing policies of philosophy journals. I know there have been calls for Brian to extend the Gourmet Report to journals before, over issues of feedback and time lags, and this discussion adds merit to those calls. If not the Gourmet Report then where?

Invitation only journals are particularly worrying if they are crowding-out open-journals but I have no idea if this is the case. Of course many books invite chapters too, although that is not to say that the submitted chapters are not reviewed.

Thom writes that as long as we are not complaining about the quality why fix what ain't broke. Well because there are issues of opportunity and equity as well as quality, although from the discussion above this seems to be orthogonal to the issue of in-house journals, which were the subject of Thom's comments.

[Christopher Pynes]

Comment on Weatherson and Sturgeon post:

Why I think the comment about Block vs. Shoemaker and Cooper vs. Irwin misses the point. Not commenting about these individuals, but imagine the following case.

Philosopher X (in house)
Philosopher Y (not in house)

Say that X and Y disagree on some issue about Aristotle or Dualism or whatever. If the journal always uses X, then the journal may (or perhaps will) have a bias. Now this could be the case if the journal always used Y as well. But the main point is that there is more to the issue than just using the good people in the department in question. Good people disagree philosophically. If journals are to avoid bias (when they can and it is reasonable), then they should otherwise things like _The Journal of Philosophy and Decision Theory_ (to quote Leiter from above) will occur. I might add that the same thing can be said of the students of such leading figures as well. If a journal just uses the family tree of Philosopher X (or Y), there very well might be a strong bias in favor of a certain method or topic as well. So, the issue could be broader than just current in house individuals. I feel that there is a similar analogy with hiring practices and affirmative action as well. If we have subtle biases and institutional bias against certain groups that are not part of the established group (in house philosopher in this case), then bias will occur even if it is unintentional.

So, the main issue is to eliminate bias (real or imagined) by having review practices that are above board when it is possible (within reason) to do so. I think this protects the process, the journals, the reviewers, and the authors as well.

[Rachel Barney]

I guess I’m not convinced that there’s a problem here. (Well obviously the kind of practices Robert Talisse reports are a problem; I mean an in-principle problem with in-house journals.) Any journal that cares about quality control will be using a fairly limited pool of assessors, themselves chosen in a non-blind way, and as Tad notes the editors or editorial board will then have to second-guess them to some extent. Over time that will tend to produce a certain set of priorities and both methodological and substantive biases. And why not? Why shouldn't one journal have tons o' Rawls, and another be known for decision theory? (It certainly saves me time in the reading room.) Every journal has some sort of personality, and an interesting, diverse department with high standards will tend to produce an interesting, diverse journal with high standards. The source of the anxiety here seems to be the assumption that there can only be two or three ‘top’ journals in the field, their names graven in stone by our forefathers, which therefore have a duty to represent the full diversity of the profession in some unimpeachably neutral way. Surely that assumption has got to go?

[Keith DeRose]

1. About the competence of refs. I assume it's my use of "the best of both worlds" that earns me the tag of having a "Panglossian read" here. But I don't see what's wrong with the reasoning involved. I do rely on the assumption that excellent potential refs are more likely to accept a job for a journal that's run by their own dept. Perhaps that assumption is being questioned? (If so, I can only suggest that one try editing a journal, and try sending lots of e-mails to recognized prominent hot-shots in various areas that you have no real connection to, asking them if they'll devote several hours of their time to a refereeing project they'll receive no compensation for, and see how often you're successful.) Otherwise, I don't see what the problem could be, given that the "in-house" journal has access to outside refs that's the same as other journals. They have an inside track on some excellent refs, and no relative disadvantage that I can see.

2. But I agree with Brian that the "the worry is less about competence of referees." (In fact, I pretty much said as much.) And I don't want to deny that there are real worries about a small group of people from one department having a good deal of power. But I do think more has to be said about what the special danger is in their being all from one department. I don't doubt that there are some special dangers here -- I'm just saying that the specific worries should be clearly identified. The editors of any journal will be a fairly small group of "insiders," it seems. What would help a lot here is if the reformers have something more to say about how things should work in these journals (esp. how the editors should be chosen), so we can have some idea of the other term of comparison here.

[Keith DeRose]

Incidentally, how *are* editorial boards chosen at most journals? I really don't know. But I've always imagined it works something like this: Somehow or other, you get a main editor or two or three. (Maybe they propose a journal to a publisher, which agrees to publish their proposed journal.) They then choose an editorial board. (Or they choose the initial members, who then suggest further members.) Is that how it works? If so, that looks like it's *more* prone to the danger of narrowness of various bad kinds than is a journal whose board consists of the members of a department.

[David Velleman]

I think it's unfortunate that Professors Dennett and Churchland treat the Journal of Philosophy and the Philosophical Review as similar solely because they are both (in some sense) "in house". To the best of my knowledge (which, admittedly, may not be very good) the editorial practices of these journals are very different. This thread has an excellent statement from the editors of the Phil. Review, which makes clear that their journal is not actually "in house" in any meaningful sense. Could someone prevail on the editors of the Journal of Philosophy to post an explanation of their editorial process?

A word about journals published by professional associations. In many disciplines, these journals are the main source of income for the associations that published them. The journals are consequently fairly expensive. A few years ago, I was told that the premier psychology journals, published by the AP(sych)A were refusing to publish work if it had already been posted on the web, with the result that psychologists don't post their work-in-progress as many philosophers do. I was told that this policy was motivated by the APA's need to protect it's source of income from being undermined by free online publication. Maybe someone who knows about psychology could confirm this?

Finally, I want to agree with Rachel Barney: I don't see the problem with "in house" journals, either. I do see a problem with one person or department editing multiple journals. But I am more concerned about the proliferation of non-refereed volumes (anthologies, encyclopedias, "handbooks", etc.) marketed primarily to academic libraries, which are already paying for subscriptions to the journals (and in many cases, paying twice -- once for the print journal and again for the online version). A library that subscribes to all of the philosophy journals ought to receive almost all of the best philosophical research.

[Tom Hurka]

Obviously, there's no simple in-house/non-house dichotomy. There's a continuum running from a journal edited by one person (Mind under G. E. Moore or Simon Blackburn?) to a journal in which decision-making is in effect distributed among several hundred referees. And on this continuum in-house/non-house doesn't matter in itself; what matters is only the concentration of decision-making power and the dangers (which I think outweigh the benefits) it can bring. A nominally in-house journal like PR can consult more referees than a nominally non-house journal whose editor mostly uses just his/her departmental colleagues.

That said, I'm not as reassured by the Sturgeon/Weatherson post as some others. They say PR now sends "quite a few more papers" to outside referees, but that suggests there's still a fair amount of refereeing by Cornell department members, i.e., a less wide disperson of decision-making than at many other journals. And, as Christopher Pynes says, even if the Cornell expert in philosophy of x is as competent as anyone else in philosophy of x, if PR uses him/her disproportionately that gives that person's tastes and assumptions disproportionate influence.

Re Rachel Barney: The names of the top journals (and by definition there can't be more than a few) aren't graven in stone but they aren't written on water either. There's path-dependence, in that a reputation as a top-rated journal tends to perpetuate itself, to some extent independently of the journal's content. Ask yourself: how much change has there been in the top-rated journals in the past 50 years? If you wanted to start a new journal to supplant PR, JP, or Phronesis, how long would it take? I think a pretty long time.

Finally, I agree with Keith DeRose that this issue about journals is less important than the one about invitation-only volumes, which was vigorously discussed in the thread he links to.

P.S. Re Keith, is JP a blind review journal? I don't think it used to be, as I believe Phil and Public Affairs wasn't (though it now is).

[Charles]

Forgive my ignorance (or naivete?), but as a student looking to enter the field, I find it hard to swallow the presuppositions of this debate.

It strikes me that if the journals in question were the *worst* journals in the field, this would be a non-issue: good philosophers would simply choose not to publish in those journals. So it seems that the issue is the interaction between the fairness of the editorial practices and the high reputation of specific journals. So let me raise the following question: if every philosopher were judged not on the pedigree of their degree and publications, but rather solely on the quality of their work itself, would this be an issue?

That is, if hiring and tenure decisions were based on concrete evaluations of a scholar's body of work and not (largely) on the socio-historical factors of that person's pedigree, wouldn't philosophers simply be able to choose not to publish in journals whose editorial practices they deemed unfit? In such a scenario, the (potential?) problems raised by in-house editing would be self-corrected if the community felt those practices to result in or potentially result in unfairness. The same applies to invitation-only journals. Insularity would be a non-issue if that insularity did not directly influence hiring and tenure decisions, as it seems to (unfairly) do.

Of course, philosophers are as imperfect as the rest of us, so sociological factors will inevitably creep in. Nonetheless, shouldn't philosophers focus on the more fundamental causes of unfairness in the profession?

[Keith DeRose]

Tom: Now that I think of it, I don't know whether JP uses blind review. It was a mental glitch that got it into my head that they did use blind review when I implied in a previous comment that they did.

[Kevin Timpe]

The Guidelines for Authors at JP doesn't prescribe authors to prepare manuscripts for blind review before submission (At least, it doesn't on their web-site. Perhaps the guidelines printed in the journal itself are different). I suppose that it is possible that they could remove all identifying marks from submissions before they are sent to authors, but I would think this is unlikely.

[Seiriol Morgan]

Aren't we missing something here? The editors of the Review have given a very clear account of their procedures for selecting referees, and the rationale for it. But they have rather glossed over another matter which should be of equal or greater concern to the circling critics, which is this: what percentage of submissions received are actually read by a referee at all (aside from the two editors, of course)? Their website says they have a two-stage review process. At stage one the submission is read by both editors. If it passes this stage then it is sent to an expert referee for further evaluation. The website then goes on to say that "the great majority" of submissions do not pass the first stage of review. They don't put a figure on "great majority", but one would assume it must be at least two-thirds and possibly a lot more. So if the statement is correct a very large number of the Review's submissions are binned without undergoing the scrutiny of an independent referee, solely as a result of the editors' judgement. (I've heard anecdotaly that if you get your paper through to the second round at the Review you've around a 50% chance of seeing it published. Since they say they publish about 5% of submissions this suggests that 90% of them are eliminated before the referees enter the picture. But I stress that this is anecdotal.)

It's hard to see why the critics should feel reassured by even the most robust process of selecting referees, if the great majority of papers get the thumbs-down before getting anywhere near a referee's desk at all. After all, the rigour in the review process seems to be geared entirely towards ensuring that they don't accidently publish a paper that's not in fact much good, rather than making sure they don't miss something that is very good, which is what Brian expressed concern about. And here there does seem to be a relevant difference wih other journals. Tad Brennan pointed out that editorial discretion must be central to determining journal content, because every top journal will receive many more publishable submissions than it has space to publish. But as I understand it the other journals send out to referees everything they receive which is not obviously a bedsit theory. So when the editors are making their decisions their judgements will be bolstered by reports on each paper by specialists in the field, and an initial impression that a paper is confused or pedestrian might be entirely overturned once it is accompanied by two wildly enthusiastic referees' reports.

So, whilst in one respect things look better than was suggested by Dennett and Churchland, in another they look much worse. For the great majority of papers it is not even that the pool of assessors is limited to the Cornell faculty. In fact it is limited to a mere two of them at any one time.

However, now that I've offered the critics some more ammunition, let me say that I'm not of their camp. Thom Brooks and others talked above about the risks of insularity with these journals. You can see why, abstractly. But to my mind there is an irony here, since in fact the Phil Review (along with PPR) is the least insular of the front-line journals, by quite some distance. I'm sorry to bring up the history chestnut once again, but which of the premier journals have published more than a mere smattering of papers on historical figures recently, even central ones like Kant and Descartes? Well, Philosophical Review has. Since 2000 they've published papers on Plato, ancient scepticism, the medievals (and their reception of Aristotle's metaphysics), Hobbes, Descartes (more than once), Pascal, Locke, Leibniz (more than once), Reid, Hume, Rousseau, Kant (several times), Kant and Frege. Kant and the Greek mathematical tradition, of all things. (Who says you can't get stuff of minority interest into PR?) All they need to do is publish something on Hegel or Nietzsche, and they'd have as big a tent as anyone could reasonably ask for. Aside from PPR, which of the others have? None of them is the answer. (Maybe PQ is slightly less remiss here.) And of course PR has been publishing stuff in metaphysics, philscience, metaethics, mind- all the areas of hot current interest- at the same time. (Roughly 25% of papers published have been historical.)

I'm all for the wide use of editorial discretion, when that discretion is used to preserve the character of a journal that plays a positive role for the profession. PR seems to consciously take itself to have a mission to present a genuine review of the discipline, publishing strong work right across the range of topics people are interested in. No-one has argued that the work published is not by and large of high quality. So they're not insular and they publish good work. They're doing a good job then. Whatever changes they might make to the review process, it would be a grave loss if this character were not preserved.

[David Velleman]

It is not realistic to expect a journal to send out every submission for refereeing. Journals receive all sorts of manuscripts that are clearly ineligible for publication for reasons that can be assessed without subject-area expertise. Recruiting referees is already difficult: everyone is busy, and refereeing can be a chore. Referees are entitled to expect that they will be asked to read only those manuscripts on which a referee's opinion is needed.

[Tim O'Keefe]

David Velleman is exactly right: reviewers shouldn't be expected to waste their time on manuscripts that are pretty plainly unsuitable. And any editors who regularly sent out marginal papers to reviewers based on Seiriol Morgan's grounds--that they were worried that their initial impression of a paper may be mistaken--would soon find it difficult to recruit reviewers.

This might result (very occasionally) in a good paper being passed over, but so it goes. I don't see any fundamental unfairness here, and you do the best you can with fallible human beings having limited time on their hands.

[Keith DeRose]

At the end of my Aug. 30 10:28 AM comment, I wrote: "So, in what seems to me the most important issue, the PR and JP are on the good side, and, along with many other journals, deserve our support." If JP doesn't use blind review, that statement should be revised to be just about the PR.

Seiriol Morgan raises an important point about the PR's procedure which I had missed. (Incidentally, the PR is admirably upfront about how it operates; see (as I have just done) its explanation here: http://www.arts.cornell.edu/philrev/info/editorial.html .)

His point reinforces Tad Brennan's reminder about the importance of editors. Not only do they choose the referees, but they actually make the final decisions on which papers go in, since the various refs are unlikely to give the thumbs up to the right number of papers. It's unrealistic to say that the journal should accept all the papers the refs recommend accepting – especially at highly selective journals. And I say this as someone who was once the victim of just the phenomenon in question. For one of my papers I received from the journal in question – Nous – two quite positive referees' reports that recommended publishing the paper, and a letter from the editor (and I don't now remember who that was at the time) explaining that as the journal was getting too many papers where both refs recommended publication, they weren't going to publish mine in light of the reservations expressed by one of the referees. (Later, I found out who that less positive referee was when he asked what happened to that paper. He was quite surprised and disappointed that it was his report that was the block.) I was tempted to be quite upset: "Well, what are referees for, if you won't listen to them, even when both are saying to publish a paper?!" But, when you think of it, what is a journal to do when they get far too many positive referee reports? The editors just have to make some decisions here. They may be guided by the referees' reports, favoring papers which receive the most positive of the positive reports. But they have to decide.

The PR is extremely selective. (My guess is that their 5% acceptance rate, low as it is, understates by quite a bit how selective they are. Not only are they highly selective, but they're well-known to be. Thus, many potential authors do a lot of "self-editing," and only send their best papers to the PR. Yet they still take only 5%.) It seems unavoidable that the editors will have to play a big role in the selection process. For the reasons David Velleman gives, it seems reasonable, at least to me, for them to play some of this role early in the process, before papers go to referees.

A possible reform is to have all submitted papers go to referees. I'm with David in not supporting this possible reform. David raises important type of consideration which is often missed by would-be reformers of journals: that refereeing is in many ways a lousy job. It's a job that's extremely important to our profession, the realization of which, I suppose, is a big reason why it's accepted as often as it is. But we have to be extremely careful about doing things to discourage referees. (This came up in an earlier on-line discussion when would-be reformers were starting to clamour to have the identities of referees revealed to the authors of the papers they judge. I objected that this would be very bad for referees: not only are they performing a somewhat onerous (at least I find it so) and thankless task for no compensation, but now they are going to start to accumulate enemies in the profession when they advise rejecting papers. Who needs that? I suspect very many would immediately stop accepting any requests to referee if that "reform" were implemented.) Asking referees to put hours into a paper that has no realistic chance of being published anyway would also be such a discouragement. And, in extremely selective journals, this needn't be limited to papers which have some obvious fatal flaw: even papers that are in many ways quite good can, for one reason or another, have no realistic chance of getting into an extremely selective journal, just as many very good running backs have no realistic chance of making it in the NFL, and will be cut from the pool before they get to the stage of being judged by the real experts.

But for those who would advocate such a reform, I *believe* you'll end up reforming lots of journals that are not of the "in-house" variety we're talking about here, as well.

To my thinking, what becomes very important in light of the power of editors is that, so far as possible (sometimes, unavoidably, expert refs will be able to make a good guess as to the identity of an author), editors, as well as referees, be kept “blind” to the identity of authors until a final decision is reached. And the PR states that that is their policy.

The power of editors can also raise issues about the likes & dislikes of a few people becoming too important. But if that's one's worry, on this score, the PR may actually do quite well relative to most philosophy journals, as the editorship of the journal rotates every 6 to 12 months. Of course, it rotates within a fairly small group of people – the Cornell senior philosophers – and that might seem far from ideal to some would-be reformers. But when you compare it with other journals, some of which have the same one editor (who of course, in a limiting sense, is in the same department as herself!) for years and years, I suspect the PR might start looking quite good by comparison.

Reformers might have some other idea of how to do things. Concrete proposals are needed before comparative judgments can be made.

[David Thacher]

A quick comment on the statement above that "no other academic discipline has first ranked journals that are edited in house": This is not actually correct for economics and sociology. The Journal of Political Economy is a University of Chicago journal, and the Quarterly Journal of Economics is a Harvard journal; both clearly rank among the "big four" economics journals (along with the American Economic Review and Econometrica). In sociology, the American Journal of Sociology is a U of Chicago journal, and it shares "top journal" status with the American Sociological Review.

Of course a major theme in this discussion has been that "in house" can mean lots of different things, and I don't know how actual review practices of these journals compare with PR and J Phil. At minimum, the editors for each journal are chosen from faculty at its home department. On the other hand, AJS, at least, has editorial board members from outside of Chicago, and it regularly consults reviewers from other universities.

[Brian Leiter]

Clearly not every submission should go to referees. A different model has been mentioned several times, however, namely, Ethics, where the Editor (currently my colleague John Deigh) and the Associate Editors (a diverse group at more than a half-dozen different departments) decide which papers should go out to referees. Ethics, in turn, uses a very large pool of referees, and is well-known in the profession, I think, for providing helpful referee reports, even when papers are rejected. Ethics isn't the only journal that is clearly not in-house (others are noted by others above), but it is one useful example.

[Rob Rupert]

I am sympathetic to Dennett and Churchland’s general point – that top philosophy journals should operate in keeping with principles of best practice similar to those used at the top science journals. Nevertheless, when estimating the need for such practices, we should look at the results of current editorial practice (a point others have emphasized). Since we don’t have in hand the right sort of quantitative data, the anecdotal evidence must do; and the anecdotal evidence to which I have access makes JP and PR look pretty good. I know of quite a few cases in which JP or PR has accepted the work of a relatively unconnected philosopher or in which a paper critical of the work of someone at Columbia or Cornell was accepted (or at least taken very seriously and nixed by an outside referee). For example, in the past five years, I seen Texas Tech at end of four JP articles, and none of the folks involved have Ph.D.s from top-ten departments. Of course Ed Averill (one of the authors in question) has a long history of publishing in JP and PR, but that fact itself is evidence that editorial review at JP and PR is functioning properly. Furthermore, I know of many cases where JP and PR have rejected papers by well-connected philosophers. So, I tend to doubt that cronyism drives editorial decisions at JP or PR.

I wonder whether any of you know of cases on the other side of the coin. The stuff about JP and decision theory doesn't seem very convincing to me (as an example of cronyism or insularity). Don't many journals publish waves of material on hot topics, and isn't this, in some cases at least, a natural and appropriate thing to do? I'm no expert on the history of DT's use in philosophy, but let's say it went this way. The apparatus of DT was quietly developed by specialists prior to the early nineties, without being applied to a wide variety of philosophical problems (and when it was used by well-known philosophers -- Popper, Carnap, Lewis, van Fraassen -- the presentation was too dense or difficult for most people outside of formal sciences to get their heads around). Now let's say that in the late 80's (maybe because of Hampton's or Kavka's work), the promise of DT was recognized by the broader philosophical community. If this is how things went, one should not be surprised by the burst in the early nineties of published work employing DT. Furthermore, it doesn't seem surprising to me that JP published a substantial amount of this work. I assume that many of the papers were interdisciplinary, and part of JP's charge is to publish work at the boundaries of philosophy and other disciplines. Furthermore, the nature of the tool (DT) is such that, if one gets results using it, they are more concrete and definitive than what one gets by applying many of the other extant philosophical tools (maybe you can _prove_ something using DT, and isn't one role of the top journals to publish the most convincing results produced by philosophers, modulo considerations of the topic's importance?).

[Tom Hurka]

Again, it's a question of degrees. Of course a journal shouldn't send out EVERY paper it receives to referees. But if Seiriol Morgan's speculation that 90% of submissions to PR are turned down before being sent to referees is accurate -- and I have no idea whether it is -- I would say that figure is too high. (This is screening by just two editors. Do any two philosophers have the specific expertise to make that sort of judgement about 90% of the papers, on all varied topics, that come into a journal like PR?) My understanding is that at Ethics, where the relevant figure varies among Associate Editors, about a third of submissions are rejected without being sent to referees, and that was certainly my average when I was an editor at CJP. But the main point is just that the fact that it would be foolish to send all submissions out to refereeing doesn't imply that any percentage above zero is OK.

[Keith DeRose]

How were the editor and associate editors of Ethics selected? I just don't know.

I ask b/c for many of the ways I'd take to be most likely, they could comprise a fairly group of "insiders" *more* worrisome than the members of certain philosophy departments. This might be the case, for example, if the assoc. editors were at some time chosen by some earlier editor, or some other small group of people, and the so-selected editor & assoc. editors then started selecting the new editors & assoc. editors as slots opened up. I stress I'm just speculating here; I now nothing of the history of *Ethics* (even less than I know of the history of ethics!). But I'm trying to stress that the fact that the editors of a journal come from different departments is not by itself much of reassurance against narrowness and insularity.

[Seiriol Morgan]

Yes, I somewhat overstated the point in my post, since I accept everything David Velleman says above. There are many good reasons why a paper might be rejected outright even if it isn't a bedsit theory: too long, off topic, elementary error immediately evident, three other papers on the demandingness objection are already lined up for the next issue etc. (I once had a paper sent straight back to me by Ethical Theory and Moral Practice: too long and too boring. It was as well, and I'm glad a referee didn't have to suffer it. The editor's candour led me to just abandon it, which was the right thing to do.)

This might add up to a lot of papers. But the great majority of submissions? After all, it isn't natural to interpret the Review's statement that there is a two-stage review process in which at stage one all the papers are read by the editors alone and the great majority eliminated as meaning "First we go through them and throw away all the hopeless cases". I assumed that this goes without saying for every journal.

So I suppose the question is: do the allegedly more virtuous journals have review procedures which could reasonably be described as the editors going through them personally and eliminating the great majority of submissions, before the rest are sent to referees? If the answer is yes then the Review has no case to answer. If no then the general worry remains. I don't know what the answer is, since I've never had anything to do with editing a journal, though of course I suspect it is no, or I wouldn't have raised the point. (Though as I said, I like things the way they are, so I would be happy to be corrected on this.)

[Mike]

Seiriol's point might be a bit different from the wildly unrealstic suggestion that every paper at PR be sent out to referees. The idea might be that if PR has an acceptance rate of 5% or less (as Keith D. suggests), and if lots of would-be authors with various specializations are self-editing (i.e., submitting their best work), then the editors at PR are doing lots of culling from an already highly selected set of submissions. The risk of a mistaken evaluation from two editors in these circumstances does seem considerably greater and worth noting.

[Seiriol Morgan]

I'm feeling somewhat nervous now about having mentioned the 90% figure, since that was something I heard in a pub, which I'm in no position to substantiate. It could well be that I'm spreading misinformation in a very public forum. Let's stick to what they've told us themselves, which is that the "great majority" of papers are eliminated by the editors before the involvement of referees. Of course, whatever the figure turns out to be it will be a lot more than the third of papers Tom Hurka says he rejected at this stage at CJP.

[Steve Hales]

I found it interesting that Brian holds up Ethics as a model journal. A few years ago, they did to me what Nous did to Keith--sent a paper of mine for review, 7 months later told me that all the referees recommended that paper for publication, and still the editors were rejecting it. They could have done that at the start and saved me 7 months of sitting around waiting. Perhaps things have changed at Ethics since then.

I think that there are many problems with the way in which journals are run--unestablished junior people invited solely on buzz to publish in prestigious invitation-only venues, the mere existence of so many invitation-only "journals," journals whose editorial policies exhibit extreme insularity or topical bias, journals with clear preferences for famous or well-connected authors, journals that take 2+ years to publish an article after acceptance, journals that routinely make authors wait for a year only to receive a rejection with no comments or unintelligible ones, and so on.

It seems to me that as a profession we should decide which of these issues are the most serious and devote our energies to solving the worst problems first. Certainly, looking at how things are done at journals in other fields for ideas is a step in the right direction.

[Colin Farrelly]

Keith invites reformers to make concrete suggestions so here is my two cents worth.

From the various comments made so far one major concern is that lots of us are in the dark. That fact alone is worthy of concern. We are in the dark concerning the journals that carry the greatest weight in terms of tenure, promotion, etc. Do submissions to JP receive blind review? What portion of submissions to Phil Review go to external reviewers?

Those defending the status quo argue the end product is all that matters (i.e. great papers are published) but I think we need to take concerns of fair procedure seriously. Especially when the journals have so much sway. The more weight the journal carries in the discipline, the more transparency (and accountability) it ought to have.

So what can be done? A partial solution would be for these journals to post a list of the number of submissions they receive each year, the sub-fields of the submissions (e.g. metaphysics, ethics, epistemology, etc.) and the rate of acceptance in each field. The Journal of Politics does this. See

http://journalofpolitics.org/manustats.html

Furthermore, in the case of Phil Review, I think they should also state the percentage of the submissions that make it past the first internal round of reviews.

And finally, and I think this is imperative, they should have a rule that all papers accepted for publication must have at least one positive external referee report. In other words, the Cornell faculty (editors and in-house referees) would not be able, by themselves, to satisfy all review requirements.

These kinds of modest changes (which I think are only a start) would create greater transparency and would not impose heavy burdens on the editors of the journals.

Cheers,
Colin

[Brian Leiter]

Regarding Steve Hales's comment: I am sure even the best-run journals will have at least one rejected author who believe he or she was treated badly in the process. Indeed, I bet the best-run journals will have more than one author, rejected or not, who feels the process didn't work well. I don't think that has any significant bearing on the main point, which is that Ethics is not an in-house journal, and it operates a generally admirable and responsible process of blind peer review.

[Mike DePaul]

One thing referees do is provide a journal editor with an assessment of the quality of a paper, most significantly, an opionion about whether the paper is good enough to merit publication in that particular journal. But another thing referees can do is provide feedback to authors. One reason to err on the side of sending papers out to referees, even when they pretty clearly will not be accepted, is that this might serve to secure high quality feedback for authors -- including the authors of papers that are not very good.

A case can obviously be made that authors can get feedback in other ways, and even that authors should have got the feedback required for really bad papers back when they were in graduate school. But the fact is, there are many people in our profession -- think, e.g., of colleagues in developing countries -- who do not have ready access to decent feedback on their efforts. Journals can provide a valuable service by serving as a go between in securing expert comments for such authors.

Now I'm not saying that this is something that all journals should to do. It surely makes sense that some journals, obviously the most highly regarded ones, primarily sort out the highest quality work available from everything else. But I think we should also have some journals that both sort the chaff from the grain and try to get all authors (whose work meets a fairly minimal standard) some high quality feedback.

Providing helpful comments is hard work, and it is usually harder when the paper is marginal. And as with all refereeing, it is, as others have noted, thankless work. But I don't think it is crazy to think that those of us who have relatively desirable academic positions can devote a certain amount of time to helping colleagues improve their work -- even when it is marginal.

Refereeing is, after all, service to the profession. There is no denying that telling a really good paper from an amazingly excellent paper is such a service. But so is explaining what must be done to turn a marginal paper into a solid paper.

[Edmundas Adomonis]

My understanding is (based, among other things, on my experience about regional philosophical journals) that the quality of a journal crucially depends, firstly, on the competence of those who referee papers and, secondly, on the carefulness of referees, their willingness to do the job properly, thirdly, on reviewing being blind. It doesn't matter whether the journal is in-house or out-house or whatever. I think the editors of the Philosophical Review explained everything all right in the comment somewhere above.

By the way, those who think that the Philosophical Review is not good enough might stop reading it.

[Gregory Wheeler]

Although referee work is onerous, I disagree with those who claim that it is unreasonable to expect a philosophy journal to properly review all manuscripts it receives, or to expect that it provide reports to authors along with the decision. Bad papers often can be quickly and constructively reviewed, and these reviews offer a direct benefit to the author, which in turn improves the health of the field.

It might be helpful to highlight a practice in computer science, which embraces the policy of providing reports to authors. Nearly all journals and professional conferences with published proceedings follow this policy.

Consider conferences. It is standard to blind review every paper, and to send two or three reports to authors along with the decision. In cases of conflicting votes on a paper, discussions are held where each referee can defend his vote. This process usually reveals who among the referees is strongest, and gives the chair important information that is helpful for resolving borderline cases. And all of this is done usually within 6 weeks of receiving between 50-200 papers, with decisions delivered to all authors on a previously announced date.

Automated systems like Confman and EasyChair make this traffic jam manageable, and the Springer Journals (like JPL, which is excellent) are using a version of this type of program, but the key to the system working are the standards demanded by the community. All reputable conferences follow these best practices because fairness, courtesy, timeliness, and transparency are demanded by those in the field.

My CS colleagues fall out of their chairs laughing when I mention philosophy journals excusing themselves from sending a referee report because they receive 500 papers in a year.

[Les Green]

On Ethics: Another of its practices that I greatly value is that of sending referee's comments not only to authors, but also to the other referees. Most journals do not do this, and so I'm sometimes left wondering what inspired the editors' decisions. With Ethics the 'thankless task' at least brings the reward of seeing what someone else thought of the piece, and sometimes it casts light on the ultimate decision whether to publish. I'd like to see more journals follow suit.

[Jacob T. Levy]

I strongly agree with Les Green's point immediately above. In Political Science the APSR does this, as does (I think) the JOP, but it's otherwise uncommon. Indeed, some journals don't even tell the referees the final decision, which I find strange and discouraging.

[harry brighouse]

Ethics is extremely upfront about the editorial process -- about the fact that it is lengthy and that even if both referees and the relevant associate editor support the paper it has only a 50/50 chance of acceptance. This makes it much easier for someone who has reasons to have a short time horizon to make a judgement about whether it is wise to submit in the first place and, after that, to persist, than with some journals which are less forthcoming.

[Chris Bertram]

Slightly tangential to the main discussion, but of interest to people thinking about these issues is

http://www.iew.unizh.ch/wp/iewwp280.pdf

by Bruno Frey and one of his co-researchers. It analyses some of the dysfunctional aspects of the revise and resubmit process and recommends a radical reform.

[Tim O'Keefe]

Mike DePaul brings up an excellent point about the value of reviewers' comments for the person who sent in a paper and says that this gives some reason to send out marginal papers to reviewers, even if their chances of getting accepted are quite slim.

I can see that, but a couple of considerations on the other side:

* If the editors reject a paper on the initial screening, the author will know about the rejection very quickly and can decide what to do, instead of waiting months and months for the reviewers to do their job. For people on the TT or trying to land a TT job, this isn't trivial. So if the paper will very likely be rejected in the end anyway, why make a person wait?

* Rejection at the stage of initial screening needn't be uninformative. Since editors (typically) aren't experts in the topic the paper is on, they probably won't give extensive feedback on the particulars of the paper's argument, use of the literature, etc. But they still have a good sense of what makes a suitable paper and what doesn't, and they can (and I think should) say, in a tactful but frank way, why the paper didn't get past the first round. This sort of feedback could be equally (if not more) valuable to people starting out as are a typical reviewer's comments. I imagine that most of the papers that don't get past the initial screening get culled for some common reason that applies to lots of papers. So editors should be able to compose stock letters (suitably modified for the case at hand) and let authors know why they didn't make it past the initial screening, in a way that isn't too onerous for them.

[paulraymont]

Re. helpful referees' comments: journals should ask referees if they're willing to be identified to the author after a verdict has been reached. It would be nice to have the opportunity to acknowledge a referee by name (instead of as an anonymous figure) for helpful tips and objections. Also, if the paper is rejected by journal A, but later published in journal B after having been revised on the basis of an A-journal referee's comments, that referee is unlikely to receive any recognition. Thanking that referee by name would offer a perk that makes refereeing - well - less thankless.

[Neil Tennant]

The American Philosophical Quarterly is a generalist journal, not in-house, focusing on problems rather than history. It publishes no discussion pieces and no reviews. It has an Editorial Board with individuals serving limited terms. The Editorship is passed on every three years.

Submissions are screened by the Editor, in consultation (on possible borderline cases) with members of the Board. Only the Editor knows the identity of the author. This cannot be avoided, since the journal has no budget. It cannot pay for the editorial assistance that would be needed if even the Editor were to be kept in the dark as to the author's identity. The Editor anonymizes the electronic file that is sent to the Editorial Board or to outside referees. (In this connection, this Editor has acquired some advanced hacking skills!)

About one third of all submissions are sent out for expert refereeing. Each paper goes to two referees. Referees are offered the reassurance that the submission has passed initial editorial scrutiny. This is an important incentive to accept the onerous task of refereeing.

Submissions that do not pass editorial (board) scrutiny are promptly rejected. In many cases, the author receives editorial suggestions for improvement before the paper is sent to referees.

Referees are asked to report within a month. Many of them do so. Some of the most famous figures in philosophy oblige with detailed reports in a matter of days; other famous figures do not deign to respond to even repeated requests for an indication whether they are willing to take on the task of refereeing. (Perhaps the latter figures do not realize how much this holds up the process of evaluation. It occasions both avoidable delays and the need for reminders. The late Elizabeth Anscombe once called it 'arrogating to oneself all the privileges of genius'. To an editor, this translates as 'being a pain in the ---'.) Referees are thanked on an annual basis, in an editorial.

Constructive comments from referees are sent to the author. Almost every paper accepted is revised in light of referees' reports. Rare is the submission that can be published as it stands.

Revised submissions are sent back to the referees for final judgment. In some cases, each referee will receive the other referee's comments from the first round of refereeing. If the author supplies comments explaining the changes made, then those are also sent to the referees. Sometimes a referee will suspect that he or she might be unduly unsympathetic, or less than ideally competent on some aspect of the paper, and will suggest an alternative judge. In such a case, we then have the view of a third outside referee.

In any case of disagreement between referees on a revised submission, the Editor has recourse to the Editorial Board, which usually has no difficulty reaching a final consensual decision. In rare cases where opinion might remain divided, the Editor will finally exercise discretion.

About one third of all papers sent out for refereeing are finally accepted. Those that are not accepted will be rejected within a matter of months.

The greatest burden on the Editor is reading submissions carefully. Since the APQ is a generalist journal, this Editor errs on the side of caution in order to avoid false negatives on submissions outside his AOSs. That is why recourse to a wide pool of AOSs on the Editorial Board is so important.

The next greatest burden on the Editor is the selection of outside referees with no obvious connection to the author. Sometimes referees will respond to the invitation by disclosing a relationship of which the Editor was unaware. Their reports can still be of great value.

The refereeing and Editorial Board standards at the APQ result, happily, in a steady state that leaves hardly any room for editorial bias in the selection of what to publish.

Having described what the current procedures are at the APQ, let me hasten to cancel any untoward implicature that the reader might draw concerning in-house journals run on a different model. *Any* journal runs the risk of both false positives and false negatives. Every one of us has his or her favorite choice of examples of each kind at leading journals such as JP and PR. The proper questions to ask, given the fallibility of any selective system, are these: (a) What is the rate of such errors? (b) What is the average quality of the papers published? (c) What aspects of system-design give at least some assurance that the rate of such errors will be minimized?

Readers of journals, and submitting authors, also need to bear in mind that editing and refereeing are time-consuming tasks, which are often thankless. If (as some contributors to this thread have suggested) the philosophical community were to use a PGR-model to re-evaluate our journals, then thought should be devoted to the following questions. (1) How might Editors be freed up from other professional obligations so that more of their time and energy can be spent making their journals even better? (2) How might referees be recognized and rewarded for their efforts, not only by readers of the journal, but also by their Chairs and Deans? (3) How might a more positive correlation between professional stature, and a sense of obligation to the profession, be effected?

Those critical of in-house journals which nevertheless manage to maintain excellent reputations should also pause to consider that many of those in-house readers are devoting so much of their time to reading loads of submissions that their own research projects do suffer. I think we comfort ourselves with the thought that, since they are at places like Columbia or Cornell, it's simply noblesse oblige.

[Samuel Rickless]

Neil,

I found your comments very helpful and illuminating. I do have a question about two of your claims: (1) that APQ lacks a budget, and (2) that the editor knows the identity of those who submit. I assume that, after subtracting production and binding costs, mailing costs, and other incidental journal expenses, there is a little subscription money left over to pay a graduate student (or other employee) to code all submissions so that even the APQ editor does not know the identity of those who submit. If this assumption is correct, then it seems to me that true blind review requires that submissions be coded. Perhaps the assumption is incorrect. But, in that case, wouldn't it be worth a small hike in the cost of subscription to pay for the coding?

cheers,

Sam

[Neil Tennant]

Sam,

Thank you for your inquiry about the APQ's budget. Happily, I know nothing about that; the Editor's sole responsibility is to select papers. All financial and other administrative matters are dealt with by the founding Editor, who is now the Executive Editor.

You suggest an arrangement whereby a portion of the journal's revenue (if it has any) is set aside to pay for an administrative assistant. The assistant's role would be to anonymize submissions before they reach the Editor. While this might serve to expunge any residual whiff of personal bias (whether through the Editor's acquaintance or through his non-acquaintance with the author), there appear to be some drawbacks in such a system.

First, it would make the rotation of the Editorship---itself a virtue, given concerns about particular individuals exerting too much control---very impracticable. The new editor would have to have his or her assistant already lined up. Submitting authors would have to communicate with the assistant, who in turn would have to pass the anonymized communications (identified, say, only by the submission number) on to the Editor. This would place a brake on an otherwise efficient operation. Submissions that end up being accepted generate on the order of sixty pieces of email each. Many of these are from or to the author, during different stages of the editorial, refereeing and revision process. One would need a full-time assistant in order to ensure that the "anonymization steps" did not result in intolerable delays and bottlenecks in editorial correspondence. And that, one imagines, would be too costly to afford.

Another drawback would be the effect of anonymization on the Editor's selection of referees. Working behind the self-imposed veil of ignorance, the Editor would incur a heightened risk of selecting as a referee a current colleague, or past co-author or mentor of the submitting author. Assuming, on the one hand, that such persons, if chosen, would feel compelled to disclose that compromising relationship, one would suffer yet more delays in securing independent reviews of the submission. Assuming, on the other hand, that they do not disclose the relationship, one has the disadvantage of a not-fully-independent report.

On balance, it seems that an Editor who is also, in some sense, the manager of the academic operation of the journal is better able to do his or her job without the encumbrance of that extra step of anonymization.

Ironically, matters would be different at an in-house journal. There, one would have continuity and tradition. Terms of employment of assistants could overlap with those of outgoing and incoming editors, helping to ease the transition. And, since in-house journals have periodic editorial meetings, the 'bottleneck problem' for editorial correspondence would be subsumed into the wider problem, for an in-house journal, of longer-than-usual delays. These delays can occur between any two of the following points (and, at a journal like the APQ, are minimized by the policies we have in place):
(1) receipt of submission;
(2) formation of a first editorial impression;
(3) rejection (if warranted);
(4) selection of referees;
(5) confirmation, from referees, that they are willing to serve;
(6) receipt of reports (often only after diarized reminders);
(7) editorial consideration of the reports;
(8) communication of a decision (reject; revise and re-submit; accept) to the author;
(9) receipt of revised version;
(10) forwarding of same to referees;
(11) receipt of referees' reports on revised version;
(12) editorial consideration of the reports;
(13) consultation, if necessary, with the Editorial Board;
(14) communication of a final decision (reject or accept) to the author.

On balance, unless there is evidence to the contrary, Editors should be trusted to do an objective job even when they know the identity of the author. Indeed, they need to be entrusted with that knowledge in order to ensure that the process of review is genuinely blind.

The advantage enjoyed by a journal like the APQ in this regard is that any Editor who does not do an objective job can be replaced. And the number of possible replacements would always be high. At an in-house journal, by contrast, the number of possible replacements would be bounded by the number of tenured members in the Department.

It might be a good idea for the APA to draw up an official code of conduct for editors of philosophy journals. Almost all the entries that this Editor can think of can be derived quite easily by reflecting on one's own experiences as a submitting author and as a referee, and then appealing to the Golden and the Silver Rules.

For the author: If one's submission is bad, one needs to be told that it is so, firmly and quickly. If one's submission is good enough for further consideration, one appreciates editorial advice as to how to make it as good as possible before it goes out to referees. Later (hopefully, not many moons later) one wants to know what the referees thought of it. One also appreciates it when the journal editor is intellectually involved in the process, and not just rubber-stamping refereeing decisions.

As a referee, one wants some assurance that one is not being sent something awful to read. (A practice of sending out every paper for initial review by someone other than the Editor would be simply impracticable; that, after all, is one of the main reasons why a journal has an Editor!) One sometimes needs reminders at busy times of year. One likes to think that carefully formulated, constructive reports will result in an improved paper. One likes to have revisions sent to one, for confirmation that one's earlier objections have been met. And one likes to be thanked for the job done.

I cannot resist responding also to John McCumber's worry that with journals as currently established, the elderly tend to be the ones who are judging the young. Philosophy has a long tradition, and the longer one does it, the better and more knowledgeable one becomes. It also takes time and hard work to establish one's reputation to the point where one can be entrusted with refereeing journal submissions. The elderly are able to excel in Philosophy in a way that is rivalled by few other disciplines. They have also amassed a great deal of experience that makes them better able to appreciate the merits of exciting and fresh ideas and arguments. As appraisers of new work in Philosophy, the elderly are just as well-positioned as, if not better positioned than, the young. Indeed, if my experience as an Editor is representative, the young tend to produce the more disappointing work. (See last October's APQ Editorial about the problem of submissions from the 'callow aspirants' in TA rooms across the country.) Younger members of the profession also tend to be of narrower specialist persuasion and competence, and more ready to provide negative assessments, as referees, of work whose merits they might not fully appreciate. To worry, as John McCumber does, about the elderly judging the young is to assimilate Philosophy to a field like Mathematics, where almost all the truly brilliant and innovative work is done in one's early twenties. But Philosophy is not like that; philosophers who are long in the tooth can sink their teeth more deeply into a problem.

[Tom Hurka]

My experience matches Neil Tennant's: on average, younger referees are tougher, i.e. more negative, than older ones. And I agree that, after an initial blind reading, submissions can't be blind to the editor. In selecting referees he has to be able to exclude the author's departmental colleagues, personal enemies, etc., etc.

[Mike DePaul]

I want to agree with Neil Tennant about two things. First, re double blind reviewing, this would be very difficult in practice because of the need for aditional staff and the problems it would cause in selecting appropriate referees. Moreover, in my experience it really would not make any appreciable difference since decisions to publish or reject are nearly always determined by the evaluations of the referees. Second, I want to pile on about something Neil mentioned that really would make an appreciable difference: responding promptly to invitations to referee. If you have not edited a journal, you probably would not realize how big a factor this is in slowing things down. If someone does not respond, how long should you wait to write again? A week or ten days seems reasonable. Then how long do you wait before you give up and ask someone else? Another week or ten days? So now two or three weeks have passed and you've got to start all over. If two or three people you invite are non responders, this can be a huge factor. And to tell the truth, I am shocked by how often people simply do not respond to invitations to referee. So please, just say no by return mail! If you do not agree to referee you are declining to help, but if you do not respond you are positively hindering the review process.

[gualtiero piccinini]

In this thread, there seems to be consensus that papers that are rejected should be rejected either quickly by editors (possibly with no explanation, but preferably with a quick explanation) or more slowly on the basis of referee reports, which should be passed on to authors for the benefit of their work. Why, then, is it so common to wait a year or so only to get a rejection without comments? In my experience, there is no correlation between how fast a rejection is and whether an explanation is given.