Is it fair that a PhD student should be allowed to review a research paper?

Question

I really can't understand why an inexperienced PhD student should be allowed to review a research article written by researchers of considerably more experience and expertise.

How do journals/editors select referees for submissions?

Are you seriously suggesting only the most experienced researchers should be allowed to review? I suspect Stephen Hawking might feel that he has other things to do than review your papers. Also, a fundamental premise of publishing is that you publish so that everyone can review it. Research with rules about who is allowed to review it is highly suspect. — MBentley, Jun 14 '16 at 14:32
@MBentley, I'm not suggesting that only professors should be reviewing but I do think the system needs to be more regulated and more formalised in terms of training the less experienced-perhaps through certifications, etc. — John_dydx, Jun 14 '16 at 14:36
Note that the majority of users of this site are PhD students. Answers up-votes and rank will likely reflect that. — Cape Code, Jun 14 '16 at 14:47
@John: I should remark that one common way for an editor to find a referee is for the editor to look through the database of authors who have published in the same journal. — Willie Wong, Jun 14 '16 at 14:48
I feel like what's really being asked is "Some reviewer gave me a crappy review because they clearly didn't understand my work and I'm upset because I'm sure it was some snot-nosed 2nd-year PhD student and my work is good, dammit." Which I completely understand.
But that means the question should really be "Why isn't there a standard method at Journals for reviewing reviewers? Why isn't there more oversight of reviews by managing editors? Isn't allowing any random PhD student to reject a paper out of hand bad for the state of the literature?" — sintax, Jun 14 '16 at 16:34
This is the world law professors live in (except there it isn't qualified PhD students but any random college graduate who couldn't figure out what to do with his life yet got high enough LSATs to make it into law school - and for the past few years, that bar is even lower). I'd say that whether it is "fair" or not is irrelevant - probably not even well-posed. What matter is if it works - and I'm not sure whether the law review example is pro- or con- that proposition. — davidbak, Jun 14 '16 at 16:49
In my experience, I started getting requests to review after I submitted my first or second paper and presented at a conference, but before I started my PhD (but after finishing BS). Obviously credentials are not weighed as heavily as personal impressions either through written work or in person (probably to the editor or editorial board). — Mario Carneiro, Jun 15 '16 at 07:52
If you got a bad review the paper wasn't as good as you thought it was. I — Stevetech, Jun 15 '16 at 18:52
Fair for whom? Fair on the students for having the opportunity with supervisor support? Fair on the students for having it imposed on them without adequate oversight? Fair on paper authors for being judged by somebody viewed as "beneath you"? Fair on authors for being judged by somebody (potentially) inexperienced? — kwah, Jun 15 '16 at 19:09
@D.W., but those responding to a book also know that they are responding to a static document. Isn't it appropriate to indicate when and how a question has been edited so that those replying to it aren't retroactively made to look foolish? — LSpice, Jun 15 '16 at 20:30
@LSpice, no, generally speaking, you shouldn't be editing the question in a way that invalidates existing answers -- if you're doing that, you haven't put enough thought into asking the right question before you posted it. Anyway, this isn't the place to debate it; I'm just sharing the general advice on Stack Exchange. See http://meta.stackexchange.com/q/127639/160917 and http://meta.stackexchange.com/q/182673/160917 more detailed discussion; I suggest you take any further discussion there. — D.W., Jun 15 '16 at 21:00
I'm voting to close this question because it is basically a rant followed by a question that is different from the title question and unrelated to the ranting. — yo', Jun 15 '16 at 21:49
Rejecta Mathematica is an open access, online journal that publishes only papers that have been rejected from peer-reviewed journals (or conferences with comparable review standards) in the mathematical sciences. — Count Iblis, Jun 17 '16 at 21:22
Yes, this is an old question, but would the next question then be: Is it fair that a PhD student should be allowed to write a research paper? — skymningen, Jul 10 '17 at 10:29

Anonymous Physicist · Accepted Answer · 2016-06-14T23:30:06.407

162

Yes, it is fair.

The quality of reviews written by highly experienced researchers is low.
Many manuscripts have obvious flaws that should be spotted by the inexperienced.
Reviewers are (in theory) supervised by the editor.
Novice reviewers are more enthusiastic than experienced reviewers.
If the paper cannot be understood by a PhD student, it will not be useful to very many people.
PhD students may have more specialized expertise because they work on fewer projects, as compared to senior researchers (per @nayrb)

Of course, it depends on the individual PhD student. Some are better prepared to review than others.

edited Jun 14 '16 at 23:30

answered Jun 14 '16 at 12:31

Anonymous Physicist

98,828
24
203
351

26

I agree with the overall sentiment of your answer, but pretty much all of your bullets are at best heuristics. Obviously not all experienced researchers write bad reviews, and there certainly are valuable research contributions that will fly over the head of an average PhD student. – xLeitix Jun 14 '16 at 12:52
3

"The quality of reviews written by highly experienced researchers is low" - from my experience this is sometimes true, but I would not take that for granted. Of course this depends on the PhD, researcher and definition of quality. – Bitwise Jun 14 '16 at 12:58
28

@xLeitix Is there a non-heuristic way to select reviewers? – Anonymous Physicist Jun 14 '16 at 12:58
@Bitwise inter-rater reliability is a measure of quality. – Anonymous Physicist Jun 14 '16 at 12:58
5

I recently reviewed a paper. I wrote 4 pages of review, about 40 questions/comments. The other review, way more senior than me, wrote 4 lines. Novice reviewers are more enthusiastic than experienced reviewers. I definetly secund – Ander Biguri Jun 14 '16 at 14:59
7

@AnderBiguri It's a little worrying that you seem to know who the other reviewer was. – David Richerby Jun 14 '16 at 15:10
1

@DavidRicherby I didn't know who he/she was, but I know it was a senior researcher, just that. – Ander Biguri Jun 14 '16 at 15:26
11

You should also mentioned that many PhD students have greater expertise in a subfield due to specialization than the majority of postdocs or even professors. – abnry Jun 14 '16 at 15:53
2

I disagree entirely with bullet points 1,5,6. – ProfRob Jun 19 '16 at 16:37

score 67 · Answer 2 · answered Jun 14 '16 at 12:19

In many disciplines of science, asking Ph.D. students to review papers is considered to be fair by the community. I can only speak for computer science here, but the following arguments may apply in other fields as well.

When a Ph.D. student reviews her/his first papers, this is typically done for his/her advisor. Advisors often check the reviews before turning them in, and thus avoid highly unfair or imprecise evaluation of the work of others.
Reviewing work is one of the best ways to get started with writing your own work. You get to know first hand what can make a paper weak and what you want to read as a reviewer. This is very valuable to the student.
A single review on its own is seldom the reason for acceptance/rejection of a paper. If there is a strong disagreement between the reviewers, then this is normally discussed in the case of conferences (which are prime publication venues in CS), or the editor has a look and will gather further information.
Ph.D. students often simply take more time to review a paper. Thus, they can find flaws in papers that senior researchers overlook in quicker reviews.

Indeed. To your list, I would add that some PhD students have already published, so they do have experience concerning scientific publication. Also, if unsupervised by advisor (point 1), the editor might know them and consider their student status while looking at their review. — Emilie, Jun 14 '16 at 12:28
Agree: the time I most spent reviewing a single paper was during my own PhD — mgalle, Jun 14 '16 at 12:59
Thanks for your comments. It is true that students do take more time in reviewing the articles. — John_dydx, Jun 14 '16 at 13:03
@Emilie +1, in my field, half of the PhD students already published an article during their Master. — gaborous, Jun 16 '16 at 12:25

score 33 · Answer 3 · edited Jun 15 '16 at 09:13

33

The assignment of referees to an article is not a random process where a computer draws a random number and matches it to "active researcher X" in the community. At least in my field the editors play a huge role in asking the "right people" to referee the article.

By making the focus on "inexperienced graduate students" you are focusing on the wrong things. An inexperienced graduate student who, nevertheless, has done research in similar fields and therefore has the expert knowledge to say something meaningful on the manuscript is certainly a much better referee than Prof. Dr. Messenberg whose recent research only overlaps with that of the manuscript in so much as that they use the same species of animal subjects.

The question you should ask is: "Should utterly unqualified persons be allowed to referee journal article submissions?" And the easy answer is, "No, since it is called peer-review not random-Joe-Schmoe-review."

You ask then: "How do we know whether someone is qualified?"

Answer: "You trust the editors; if you don't, don't send the manuscript to him/her/that journal."

As an aside: there are two ways that I've seen where Graduate Students come to review a paper.

The graduate student is passed the paper by his or her advisor to referee. Good advisors will only do so for appropriate papers, and may even provide guidance on how to referee a paper. Bad advisors are, well, bad (if an advisor gave an unsuitable paper to a student to referee, do you really think he himself will write a good referee report?)
The graduate student has independently came to the notice of the journal editor due to a previous interaction (conference presentation, paper submission, research discussion), and the journal editor feels that the student can fill the duties of the referee.

edited Jun 15 '16 at 09:13

Cape Code

27,102
8
98
151

answered Jun 14 '16 at 14:19

Willie Wong

11,052
1
45
69

2

Exactly! If you submit a paper to a journal where you don't trust the editors to choose good referees, then you are doing it wrong. I can't imagine any plausible scenario in which you would trust the editors' judgment with the one exception that you fear they might choose a graduate student. – Anonymous Mathematician Jun 14 '16 at 14:43
I agree with the points you make, but, out of curiosity, how would you address the same question for conferences? E.g., in CS, which is mostly conference-driven. Since the PC changes every year, the "trusting the editors" parts is a much weaker point (or so it seems to me). – Clement C. Jun 14 '16 at 16:57
1

@ClementC. : I would shake my head in your general direction and give you my condolences, while secretly having a bit of schadenfreude at your expense. Or more seriously: I have zero idea how conference-driven fields like CS are run. But if a field has systematic issues where one cannot necessarily trust the conference chair (is that the right term?), that issue should probably be addressed as a higher priority than the symptom of whether graduate students get to review submissions. But that's just my knee-jerk reaction, since I have no experience nor expertise to answer your question. – Willie Wong Jun 14 '16 at 18:17
Fair enough (for the sake of clarity: I do not claim or even mutter there is an issue; only was curious as of how your argument would carry). Additionally, I had no clue what "schadenfreude" meant prior to your comment. – Clement C. Jun 14 '16 at 18:23
1

@ClementC.: I'm so happy to be able to introduce you to Avenue Q. Yes, you're welcome. – Willie Wong Jun 14 '16 at 18:26
3

@ClementC.: To mirror the words of the answer: In CS, the conference chair is not a "random Joe Schmoe", either, nor a lonely master who makes all decisions single-handedly. Rather than that, there is a program committee, which is typically populated by more or less experienced researchers with strong connections in the community; the "higher" the conference, the more well-known will various of the program committee members be. Hence, there is not really a trust issue there, either. – O. R. Mapper Jun 14 '16 at 18:49
In the journals I use, a referee is not free to pass the paper to anyone they feel like. The paper is confidential. I cannot believe it is different for any journal. The supervisor may suggest their student as a replacement reviewer to the journal editor. – ProfRob Jun 19 '16 at 16:40
@RobJeffries: it is not unheard of for advisors to "subcontract" the referee work to their students. Whether this is always done with the approval/consent of the editorial board is unknown to me. – Willie Wong Jun 20 '16 at 13:27

score 5 · Answer 4 · answered Jun 14 '16 at 12:58

Of course it is great practice for the student (critical reading improves writing as other pointed out).

But it is also fair for the authors of the paper. In general PhD students, specially towards the end, are (or should be) highly specialized in their specific (sub-) subfield and should know the literature better than anybody else. While this does not make them apt to review any paper, they are often the best person for reviewing something very related to their field.

score 5 · Answer 5 · answered Jun 14 '16 at 17:54

Why isn't there a standard method at Journals for reviewing reviewers?

There are some standard methods at some journals (e. g. keeping track of response time and such) and also there are the editors who keep track of the quality of reviews. Any standardized method on reviewing quality of reviews will most likely be flawed, however.

Why isn't there more oversight of reviews by managing editors?

I don't think that this question is based on a right premise.

Isn't allowing any random PhD student to reject a paper out of hand bad for the state of the literature?

Reviewers do not reject papers. Reviewer suggest rejection. Editors reject papers. Editors don't select random PhD students.

Why isn't there any formal training required (e.g. certification) before a less experienced researcher is allowed to review journals?

Who would you like to grant a reviewer certificate? Experienced and reputed researchers probably. These researchers should be able to judge the quality of reviewers and then decide which reviewer could be qualified to review a paper and should choose stronger reviewers for more critical/potentially ground breaking papers and less experienced reviewers for more "standard" papers. Oh wait, that's exactly what editors actually do these days.

score 4 · Answer 6 · answered Jun 14 '16 at 13:35

The title and body of your question are asking different things. The title asks whether it's fair for PhD students to review research papers, which I think is certainly the case.

The alternative would be for full participation in the research community to require formal certification. For example, should a figure well-known in a field, with decades of experience (say, as an engineer), be refused permission to fully participate in the research process simply for not having a PhD? Otherwise, would they cross some arbitrary threshold of "experienced enough"? What if that figure used their savings to self-fund a PhD? Would any existing permission be revoked, since they're now a student? What if they abandoned it half way through, would they go back to their prior status? These kind of issues seem like an unnecessary administrative burden, for a restriction which needn't exist, and would seem unfair to me, so in general I would say that it's fair for PhD students to review research papers.

The body of your question asks something different: whether inexperienced PhD students should be allowed to review research papers. I think the other answers tackle this well, but I would also say that performing reviews is one way of gaining experience.

For the record, I am currently a PhD student, I was following work in my field for years before I started, and I have performed reviews.

When I got my first review to do (as a fresh PhD student a few months after graduating with my Diplom [Master]) my professor said: "Well, your Diplom means that you are a qualified professional with all professional rights and duties as a chemist." I'd say the Diplom or Master is a quite formal certification... — cbeleites unhappy with SX, Mar 20 '19 at 22:04

Ian · Answer 7 · 2016-06-15T13:46:02.290

You are asking:

Is it fair that a PhD student should be allowed to review a research paper?

Yes:

It should be checked that a paper is written in a clear enough way that anyone with a reasonable background in the research area should be able to understand the paper. A PhD student is more likely to pick up “assumed information” that should be included in the paper, but is not, then someone that is a leading researcher in the given subfield.
A Phd student can also check that the “logic” in the paper is correct etc, and is more likely to rework all the proofs then someone with less time.

But if you had asked:

Is it fair that only PhD students should be reviewing a given research paper?

No:

It needs an expert in the subfield to know all the past papers on the subject.
It needs a long standing expert to know how a paper relates to what is being done in other fields.

Therefore a mix of reviewers is needed.

Would the down voters like to comment, as my answer seems to be getting as many down votes then upvotes. — Ian, Jun 15 '16 at 13:45

score -1 · Answer 8 · answered Jun 15 '16 at 16:34

Your question is simply ambiguous. You ask:

why an inexperienced PhD student should be allowed to review a research article?

This can be read as one of the following two different questions:

"Why an inexperienced researcher (that is by chance also a PhD student) be allow to review?"; or
"Why a PhD student, which I assume in advance to be inexperienced, because all PhD students are inexperienced, be allowed to review?".

For the first interpretation, the editor or program committee should take the inexperience of a reviewer into consideration. For the second interpretation, it lies on a false premise, and so is wrong.

Is it fair that a PhD student should be allowed to review a research paper?

8 Answers8

Linked