33

In this forum, I am reading this great question (What is required of a mathematics referee?) by a user with the name mathprofessor. There is an answer by a user with the name Buffy which starts with:

Sorry, but if that's all you do, then your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error.

I am wondering now: How exactly can somebody's career as a reviewer end after some paper they reviewed is revealed to have an error?

Sure, the editor who assigned the reviewer may not assign them ever again, but how exactly are other editors (maybe from different journals) notified not to take them as a reviewer ever again? Is there some way the editor who knows is allowed to reveal the reviewer's identity? Or some higher authority they can talk to? Or how does that work out in practice?

Let us assume the following: If the answer is field-specific, let us assume we are talking about math. Moreover, as in the other question, let us assume there is no fraud going on -- the author made a honest (but big) error in the paper, and the referee was too sloppy in their report and did not note the error.

Additional question: Are there known cases where reviewers had to end their reviewing career because they did not notice an error? Again, I am assuming no fraud is going on.

Edit: I want to say that the user with name Buffy edited the answer in question and made a much weaker claim. This solves my confusion. Thank you very much, Buffy!

  • 11
    Actually, my intention was just that the editor (and that journal generally) won't come back to you. There is no pillory and no tar bucket with feathers. You just sort of disappear. And the answer was specific to math and similar things as you note. It might be different for lit criticism, though I don't know that. On the other hand, it is possible that errors are missed even with due diligence. That is why more than one reviewer is typically used. So a single error isn't necessarily review-career ending. But review in math is much more than copy editing. – Buffy Jun 06 '19 at 21:38
  • 4
    The other question was about the actual requirements of the job of reviewing not about making a single error. The short "shock headline" quoted here doesn't capture what I actually said about the nature of the job. Please read it all before you jump to unwarranted conclusions. – Buffy Jun 07 '19 at 00:50
  • 2
    Dear @Buffy, it's nice to hear from you and thank you for your explanation! I do, however, not understand what you mean with "please read it all". I read it all and I saw nothing more related to the "shock headline". That's why I asked. –  Jun 07 '19 at 07:03
  • 11
    @Buffy As pointed out in the comments to one of the answers below, your answer contains the phrase "your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error" (emphasis mine). That does indeed imply that a "single error is necessarily review-career ending". If, as you say here, your intention was just that the editor and that journal won't come back to you, and that it isn't about a single error, you may want to consider altering that sentence. – JBentley Jun 07 '19 at 13:10
  • 11
    @Buffy Also I would point out that when it comes to written communication, the intended meaning is not anywhere near as important as the meaning most likely to be interpreted by the reader. Clearly that answer was confusing enough to generate this question by the OP, and other answers and comments seem to concur. – JBentley Jun 07 '19 at 13:15
  • 1
    Also, to emphasise how much of a nonsense it is to claim that one's reviewer career may end: plenty of journal articles in all areas contain minor/medium error. If the initial statement were true then this wouldn't be the case, clearly, and we would have run out of people doing journals reviews. – gented Jun 08 '19 at 16:48
  • 1
    @gented: Well, Buffy's statement could mean that careers were over only for major errors. Indeed, I would hope that in other areas were errors could mean that lives/millions of money were lost, after such incidents, anonymity could be removed and bad reviewers punished. –  Jun 08 '19 at 17:13
  • @user109595 In all honesty nobody in this world puts money, development or lives at risk after one paper. Things take centuries before being accepted by the community :). – gented Jun 08 '19 at 17:54
  • 1
    @gented: Of course you are right, but just for fun: Some of my professors suggested that they had their hard grading policy because "you will have a lot of examination-like situations in real life, where one small error will mean thousands of deaths";) –  Jun 08 '19 at 18:53

3 Answers3

59

No, the answer is simply that @buffy is wrong. In reality, editors work hard to find anyone willing to review a paper. They will be hesitant to exclude anyone. In most fields reviewers are anonymous, so only one publisher will know if a reviewer does a bad job.

Anonymous Physicist
  • 98,828
  • 24
  • 203
  • 351
  • 3
    But they won't come back to you for a second chance if you don't actually verify the mathematics to the best of your ability. And they assume that if you take on the task that you sort of agree that you have that ability. Copyediting can be done by others. It isn't the real job of the reviewer. – Buffy Jun 07 '19 at 00:20
  • 4
    And please read the other post and my answer before you claim that "simply" I am wrong. The other post wasn't about making a single error, but about treating math reviewing as a copy editing task without really understanding the paper being reviewed. We really really want published math papers to be actually correct. The job of the reviewer is to assure that to the best of their ability. – Buffy Jun 07 '19 at 00:40
  • 32
    You said "your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error" then below that you said "If you do an inadequate job for an editor, then you are likely done with that journal. Others won't know," The second statement is partly correct, and the first one contradicts the accurate part. – Anonymous Physicist Jun 07 '19 at 01:08
26

The answer-fragment OP quoted is incorrect. Making a mistake while reviewing is not likely to lead to the end of your reviewing career. This comes from several angles (we neglect the possibility that the editors are hard-pressed to find reviewers):

  1. In most fields you can't realistically be expected to verify everything in the paper yourself. For example if you receive a paper about a new discovery at the Large Hadron Collider, you can't be expected to build your own Large Hadron Collider, run the experiments yourself, and verify the discovery. It's simply not possible.

  2. In most fields, there is some level of good faith assumed between the authors and the journal. The journal will not assume the author is actively attempting to deceive them (until proven otherwise). They will assume the author did perform the experiment. Therefore if you accept a paper that turns out to be a fraud, nobody is going to hold it against you.

  3. Finally, only the journal that you review for is likely to know your identity. No other journal will know (unless you go public). It's possible editors will tell each other not to invite a certain reviewer, or perhaps if they are sharing the same reviewer pool, but there's no central repository of "bad reviewers" or anything like that.

In practice you'll only start receiving fewer reviewer invitations (i.e. reviewing career ended) if:

  1. You retire or pass away.
  2. You make it known that you're not reviewing anymore, e.g. with a notice on your website.
  3. You become research inactive, e.g. by not publishing new papers for a while.
einpoklum
  • 39,047
  • 6
  • 75
  • 192
Allure
  • 127,528
  • 50
  • 325
  • 493
  • 8
    And none of those three options guarantees you won't. Indeed, I doubt even hitting all three isn't enough to stop reviewer invitations. – Peter Gerdes Jun 07 '19 at 00:13
  • 4
    That's why I wrote "fewer reviewer invitations" =) – Allure Jun 07 '19 at 00:17
  • 3
    Please. The other question wasn't about making an error. It was about taking a casual attitude toward the job. Reviewing in mathematics is not copy editing. If you take a casual attitude about the job then you won't be invited back to that journal. Same for the next journal who is unlucky enough to send you a paper. This question is not the same as the earlier one. – Buffy Jun 07 '19 at 00:18
  • 7
    Shit. Guess I just killed my stack exchange reviewer career since that was a clear mistake on my part :-) – Peter Gerdes Jun 07 '19 at 00:21
  • @Buffy, don't blame us. Blame the OP who characterized your post. While I agree that you might get less invites from that journal if they realize you are taking it casually truth is that it would probably take a long time before anyone noticed that even on the good papers u were slacking.

    Indeed, Journals I review for go out of their way to instruct you to skim and if u favor reject just say 'not good enough' and stop wasting effort/time. But if you are really unserious eventually they may ask less...but so what?!?!? Pissing ppl off isn't good but you don't put reviews on a CV.

     – Peter Gerdes Jun 07 '19 at 00:31
  • 1
    @PeterGerdes, bad papers are often easy to recognize. Really good ones also. It is the in between ones that are hard. Maybe they say something useful and maybe not, but the devil is in the mathematics. But if someone isn't willing to actually do the math then they shouldn't review math papers. It explains, sadly, the problems you mention in your answer here. – Buffy Jun 07 '19 at 00:36
  • 1
    @Allure Buffy's answer is incorrect. And then you go on talking about "most fields" when Buffy's answer is to a question tagged mathematics. – sgf Jun 08 '19 at 12:38
  • 1
    @Buffy Does anyone actually do the calculations in the papers? I don't have an age long experience but for what I have seen (in different universities/countries) nobody does. – gented Jun 08 '19 at 17:49
  • @gented, in math, more is involved that just what most people think of as "calculations". Proofs can be deep and subtle. The authors work to get them right (in the absence of fraud or incompetence) but mistakes can be made. But the job of the reviewer is to assure the editor and other mathematicians that there has been an independent verification. Otherwise nothing can be trusted and every reader then needs to reconstruct everything. – Buffy Jun 08 '19 at 18:09
  • 1
    @sgf, yes, my answer and the original question is very restricted to mathematics and to very similar fields. I'm sure that almost nothing I said would apply to papers in literature, for example. But in some parts of philosophy, for example, the need is similarly critical. My daughter's epistemology dissertation made and proved certain unexpected and quite interesting claims. But her proofs needed verification almost as strong as a math paper. – Buffy Jun 08 '19 at 18:13
  • 1
    So yes, @Allure, claiming that I'm "incorrect" because something I said about mathematics specifically doesn't apply to high energy physics or "most fields" probably requires a correction. In mathematics "good faith" abounds, but it isn't the same as proof. – Buffy Jun 08 '19 at 18:39
  • @Buffy yes, of course, I know, I am just claiming that reviewers don't do that even in mathematics :). – gented Jun 08 '19 at 18:40
  • 1
    @gented, then they aren't doing a proper job. This is giving me a giant sad. – Buffy Jun 08 '19 at 18:43
  • OP did not mention "Buffy" or "Buffy's answer", and that may confuse people reading your answer. Edited. 2. Your examples from Physics are less applicable to math. There are no experiments to reproduce, plus, the assumption of good will does not mean the authors don't have some hole in their proof. Still, it can be extremely hard to locate such a hole.
    • – einpoklum Jun 10 '19 at 08:43
  • 1
    @einpoklum: OP did indeed mention Buffy. – user111955 Aug 20 '19 at 14:31