Questions from a New Journal Referee

Another young philosopher writes:

I'm at the early stage in my career where I'm just starting to be approached with some regularity to referee journal submissions. So far, I've been asked to referee four papers; I accepted all four invitations, and rejected all four papers. In three cases, the papers struck me as quite bad -- I would have considered them substandard work for a mid-level graduate student. (The fourth appeared to me to be merely seriously mistaken.)

I'm starting to wonder, now, whether I'm judging too harshly. Am I just not very good at recognizing philosophical merit? I know that four isn't a very big sample size, but I'm starting to wonder whether the problem is me. Or are most journal submissions really that bad?
I'd be interested to hear discussion from more experienced philosophers about any of the following questions:

Roughly what proportion of papers you referee are accepted, rejected, or returned for R&R?

As a younger, less-established member of the profession, am I likely to receive disproportionately low quality work to referee? (Maybe editors send papers by better-established authors to better-trusted reviewers?)

Is there any good way to calibrate my reviewing process? Any tips about how to make sure I'm treating papers fairly?

What standards do you use for judging papers? Might you recommend acceptance even if you thought there was a decisive objection to the central claim? Under what circumstances?

Do editors keep track of who the good reviewers are? How do they judge them?

Signed comments only on this thread; post only once.

Comments

[D C Cramer]

As an editorial assistant for an ethics journal (www.ethicsandmedicine.com), I can tentatively answer some of your questions:

1) The slight majority of submissions to our journal are rejected. The next most frequent recommendation is to return for R&R. Only a small minority are accepted as is. Rejecting 4 of 4 doesn't seem too atypical.

2) Journals appreciate critical reviewers. If you are overly critical, your recommendation may be balanced by the other reviewer. However, there is nothing worse than receiving a light review of a bad paper. It puts editors in a tight spot.

3) In terms of standards, if it wouldn't be an A grad paper, reject it. Even if it would be a good grad paper, if the argument is flawed or it doesn't contribute to the field, it probably shouldn't be published either.

4) I keep track of good reviewers, judged mostly by timeliness and quality of comments, not necessarily by the decisions they make on the submissions they review. I choose reviewers based on their specialty on the topic of the submission, not on their relative seniority. Sometimes good reviewers still receive bad article submissions and vice versa.

Hope this helps a bit.

[J. Bogart]

I do not see why "seriously mistaken" would be a proper basis for rejection of an article, unless the journal is the publication of a particular party. Shouldn't the assessment be based on quality of argument, not correctness?

[Les Green]

I do quite a lot of reviewing in small field. Looking back over a few of years of files, my own rates seem to be, very roughly: Reject 75%, R&R 20%, Accept as is 5%. I don't have records of how many of these ended up in print. A run of 4 or 5 clunkers in a row doesn't strike me as improbable.

My modal reasons for outright rejection seemed to be: (a) Too-long-and-too-wrong. (I work in philosophy of law, where one often sees lightweight points surrounded by 50 or 60 pp of fluff, nicely spangled with dozens of decorative footnotes.) (b)Ignorance of the literature, leading to re-invention of the wheel, and sometimes re-invention of the flint knife; (c) Writing so poor that the argument is unintelligible.

On 'decisive objections': I've accepted papers I think utterly misguided and liable to decisive refutation, provided the author is aware of the objection and makes a decent (and original) effort to answer or deflect it.

[John Basl]

As a graduate student I've only had the chance to referee one article. Luckily a faculty member offered to look at my decision and comments so I had some guidance. Even before that, I looked at a few journal issues and skimmed some articles to get an idea of the quality of the articles they published.

I would think that if the articles you decide to reject are significantly worse than the articles typically published by the journal in question, you are probably right to reject them. If you read through the journal and think "I would reject all these articles", you should probably reconsider your refereeing rubric.

[Chris Pincock]

In my somewhat limited experience, the quality of the papers I have been asked to review has gone up over time. I also think editors of better journals come to recognize certain people as 'experts' a bit later than weaker journals. So, you can expect to be asked to review better papers in your area as more people come to know what your area is. In general, though, it is not harsh to accept only 1 in 4 or 1 in 5 articles that you review. Most quality journals have an acceptance rate below 20%.

[Jack Russell Weinstein]

Many journal submissions are bad. This is in part because many journal submissions are "in process." While the Leiter Blog focuses almost exclusively on the top schools, most philosophers (like me) are in smaller schools with small philosophy faculties. Our colleagues rarely do what we do and we therefore have a very small pool of people to turn to for advice. (Not to mention that fact that most of us rarely teach in our specialization or teach grad students, both of which assist us in our research.) Conferences and e-mail relationships help, but everyone is very busy and it's hard to find time to look at others' work or get people to do the same. Besides, "big shots" tend to get help from big shots, "small shots" have to rely on other small shots. The profession is very much tracked in this manner. This means that resources, opportunities, and publication experiences is heavily weighted against those in the smaller school. Finally, most people read papers "to reject" them, not to give the author the benefit of the doubt. This is the nature of our profession. We rarely take texts at their best but look for holes, errors, contradictions, etc. and exploit these weaknesses.

So, I think it's important to remember that for many of us, the journal referee is the best feedback we get, and it will take a few submissions at least before our papers are good enough to publish. While referees shouldn't compromise standards we should read to assist not to attack. We should see our reviews as help for the future, and not condemnation of the work done. As the referee you are contributing to the next draft. You are not "the gatekeeper" of the journal.

This means, above all else, be sympathetic and kind. Offer critiques in pedagogical and collegial terms, not in a belittling or combative tone. Some of the best feedback I have ever gotten, I had to dig out from nasty diatribes authored by insulting (and sometimes politically motivated) referees. So, when I was done, I was so angry that instead of wanting to thank them for their help, I wanted to kick them for being so nasty. This too, I have found, is all too common, and has to stop.

So, in short, trust your standards, but remember that they (and you) only get better when you teach (and learn).

One final note: I do not think it is correct to reject a paper because you disagree with it. The question is not is the position "badly mistaken" but, rather, is it a reasonable interpretation supported by the argument and evidence, and would it be acceptable to a group of active scholars in the relate field. So few of us agree on anything that expecting agreement in a journal article is unrealistic.

[Mark van Roojen]

Calibration is hard and I think most people worry about it if they referee much. I find that writing comments helps me figure out what I really think about a paper. I've changed my mind in each direction more than once in the process of writing up my rationale. I also know I've made mistakes, such as not getting the point of what turned out to be an important paper.

Four for four rejections is not out of line with the range of possibilities given the relatively high standards and rejection rates in philosophy. I've gotten letters from editors reminding me of the rather high (above 90%) rejection rate in journals I have been asked to referee for. I'm assuming not every paper gets sent out to referee, but still even if only half get sent out you could easily hit half a dozen or more before you got a winner.

One really good check on one's own judgement is to referee for one of the journals that sends you a copy of both the editorial letter to the author and the other referee's comments. Ethics does this pretty regularly, though recently I haven't been getting them as regularly. Phil Quarterly has sent me the letter to the author in the past, but not every time. I find it quite helpful to see what the editor thinks of my comments and also whether the other referee and I are close to one another in our opinions.

[Original Author]

Thanks for the helpful advice, all -- this has afforded a useful perspective.

Let me add one bit of clarification: the paper I described as "merely seriously mistaken" wasn't just wrong about its conclusion. Actually, I think it was probably right in its central claim. It was seriously mistaken about the views of the author to whom it was offering a response.

[Carrie Jenkins]

I've recommended rejection for most of the papers I've been asked to referee. (I would guess I've refereed about thirty papers by now, and recommended outright acceptance once, R&R three or four times.) I don't think I'm especially harsh, but I'm finding the average quality papers I get sent to referee is getting better over time. It seems from my experience and from talking to others that (some) editors tend to send more junior philosophers the papers that they consider to be likely rejections, so as to try and ensure that the ones that they think might well end up in print will have been scrutinised by a more senior person before being published.

I agree with other commentators that thinking a paper's thesis is false is not a good reason to recommend rejection.

[Eddy Nahmias]

Like teaching, refereeing papers is an essential task in our profession, and like teaching, we rarely explicitly train our graduate students to do it. However, I adopted a practice from Guven Guzeldere (who got if from someone at Stanford, perhaps Fred Dretske?) of having students in my grad seminars write referee reports on each other's papers. They write complete drafts of their papers about a month before the end of the semester (this has its own advantages), swap papers (as I assign), and write a 3-5 page report on the paper as if it had been submitted to a real or fictitious journal or conference.

The reports must include a lengthy summary of the paper and its argument (just this can be very helpful for the authors to see), an overall assessment of the strengths and weaknesses of the paper, including a summary judgment (accept, R&R, reject), and a detailed list of specific suggestions (plus I have them give each other line edits). I give them a handout with specific guidelines and suggestions for referee reports and an article on writing reports that was published in American Phil Quarterly (I don't have it handy), and I discuss the process with them. The students get practice writing reports and feedback from me on how to do it, and often they get useful feedback from each other. I give them extended comments on the paper draft to improve their final paper ... and occasionally I also have to tell them to ignore mistaken advice from their peers.

I try to follow my own guidelines for referee reports, but sometimes I'm not so conscientious.

[Christopher Pynes]

I do something similar to Prof. Nahmias, but with undergrads. Instead of accept/R&R/reject, I have them assign grades to the papers and I give guidelines too. But if I were to make one suggestion, it would be to make the process blind to the students this way no one student knows the identity of the other person they are reviewing -- just like in real life. I also give grades on the quality of the review, and have found that student papers are much better if they know one of their peers is going to read it.

I also have rejected every MS (four now too) that I have reviewed, and two of them I thought shouldn't have gone out for review at all. But then again, editors make promises and sometimes have to have a good reason to say no to someone. In both the bad cases, I sent the review back in under three weeks.

[Derek Bowman]

Prof Pynes,

Blind peer review for students sounds like a good idea for your undergraduate classes, but most grad seminars I've been in are too small to make such blindness much more than a pretense.

[Eric Schwitzgebel]

I agree with the general thrust of the conversation so far. I would add, however, that my refereeing standards vary with my perception of the selectiveness of the journal.

[Allan Hazlett]

I've encountered tons of anecdotal evidence that young referees get lower quality papers than older referees. (E.g. many senior-level philosophers say that they didn't start getting decent papers until they were many years into their careers.) In roughly three years of refereeing, I've recommended rejection for around 15 papers, and never recommended acceptance. (I recommended my first revise and resubmit a few months ago.)

In a world where there is far too much garbage published, tough standards should be celebrated.

Although it seems a mistake to reject a paper just because its thesis is false (by your lights), we've all encountered much worse behavior from referees. Anyone who is even putting this much thought into their refereeing is ahead of 90% of referees out there.

[Kathryn Norlock]

Calibration is indeed tricky. I find that whether written responses are required or not, writing out my observations is essential to articulating my reasons for revision or rejection. Only after putting pen to paper or fingers to keys do I get a clear sense of whether I'm trucking toward an overall positive or negative review. Sometimes, e.g., mere stylistic flaws are excruciating to me after a point, and can taint my reception of the quality of the insights. Writing out, "This style annoys me," forces me to think more substantively. I'd go so far as to say that rejections without written reasons are professionally naughty at least, and morally wrong at their worst.

But to the OP's more pointful questions: Indeed many journal submissions are poor after the authors have revised and rearranged sentences until they went blind. Further, I do get the impression that we more junior scholars are likely to referee junior papers, perhaps because we tend to surface as each other's commentators and conference panelists in self-selecting conferences and emerging subfields.

Four poor papers is not necessarily a reflection on the referee, and it sounds like your reasons for finding papers quite bad are clear -- except that I spin my wheels over the question regarding "a decisive objection to the central claim." This sounds more like a disagreement in philosophy than a reason to reject a paper, and doesn't state whether the case for the central claim was presented well or ill. At such times, I ask myself if the paper ought to be rejected, or if it's rather that I disagree so much that I ought to just write an article refuting the one I'm reading. My greatest struggles lately are with those occasions on which I believe an author is misreading a source, for instance, when they demonstrate genuine misunderstanding of text (omitting a negation, for example), versus holding an interpretation that makes me grit my teeth.

The Philosophers Anonymous blog currently features a post ("Floodgates") asking how many articles, etc. one ought to referee a year. I would like to know the answer to that one myself.

[Thom Brooks]

I agree with most of the above except for the view that publishable papers are papers that deserve an A. In my view, the standard is whether a paper makes a contribution to its field. There are plenty of files floating around the Internet that say well what has already been said before. Publishable quality is found in work that does something different.

[Tom Hurka]

It was my experience -- or is my impression -- that younger philosophers are tougher referees. They have an idealized picture of how good a publishable philosophy paper should be, while older philosophers have been mugged by reality.

When I was a journal editor and had a submission I thought deserved publication, I *wouldn't* send it to two junior referees -- too likely to get trashed.

That said, if a journal has an acceptance rate around 10% (very common), and sends 2/3 of submissions to referees, it will end up accepting one in 6-7 submissions, which makes 4 reject recommendations out of 4 perfectly reasonable. Though would a more senior and maybe more relaxed philosopher have reached the same conclusions?

(Obvious point: the above claims are only about tendencies. Some younger referees are generous, and some older ones very tough.)

[Peter Smith]

Journals differ widely in their practices. In some cases, more or less every paper submitted is sent out to review. In other cases, as with ANALYSIS when I edited it, the editor (or a small in-house group) does a preliminary read, and only sends out to review the papers that look good enough to be serious contenders. If you are reviewing for a journal with the first kind of policy, then you should expect to recommend rejection of the large majority of what you see. If you are reviewing for a journal of the second kind, then you are only being sent pieces that have already jumped perhaps quite a high hurdle, and you should expect to recommend rejection of fewer.

I would have thought that it would be good practice for the editor to tell referees which sort of game they are playing. E.g. say "It might be helpful for you to know that only about 15% of papers sent out to review are accepted"; or "This strikes the editorial team as potentially publishable, but we need a more expert view: what do you think?".

And if you are in doubt, ask the editor! - speaking as an (ex)editor, we are only too grateful for conscientious and helpful referees.

[Berit Brogaard]

If you are very early on in your career, you probably shouldn't have been asked to referee a paper in the first place. Were the papers explicitly in your area? If not, then you should have declined the invitation. At any rate, if you reject a paper, then I think it's appropriate to write a long report pointing out the reasons for rejection. The editors can then decide whether or not to share the report with the author. The bad papers you received probably were student papers. Too many students submit papers to top-ish journals without consulting with their supervisors first.