Inability to review a conference paper

Question

I am reviewing a paper for a conference. The paper contains a lot of work, and from a high-level perspective seems good.

But when I go into detail I can't actually parse the paper. I don't understand many things, even if the paper is within my field. I am a 2nd year PhD student that writes papers within the field. The paper is very close to my research topic.

Some of the causes that block me from correctly reviewing the paper might be:

My impostor syndrome when seeing overly mathy papers.
Their math formulation is not the traditional one and they do naming and formulation variations. Even though they can be properly defined.
Over relies on the appendix to understand the content of the main body of the paper.
The paper content is very dense.

I have spent already 3 days reviewing and I am not understanding it. The paper might be good/bad but I can't tell.

What review should I give? Push for acceptance, even if I can not tell if the paper is flawed/bad/good? Push for rejection when the problem might be that I don't understand it myself?

One thing to note here is that arguably, as a 2nd year student still "trying" to publish work in the same area, you are way too junior to be an official reviewer (there is nothing wrong with helping somebody more senior review in your career stage, but then the answer becomes "tell this person that you have troubles understanding the paper and ask them for advise"). — xLeitix, Jun 28 '22 at 11:45
I guess that when you start getting papers accepted they invite you to be part of the reviewing process no? — Carlos Mougan, Jun 28 '22 at 11:49
Without knowing the subject area, hard to tell. In some areas three days to understand some math is too short. How much time is alloted for the review? Who advised you to accept the review assignment? — Terry Loring, Jun 28 '22 at 13:25
@TerryLoring you don't choose the paper you review, it's automatically assigned based on your publication — Carlos Mougan, Jun 28 '22 at 13:35
I am not asking why you reviewed this paper. I am asking why you agreed to review any paper. Is it standard in your field to review papers when you are an unpublished graduate student? — Terry Loring, Jun 28 '22 at 13:37
I have published in similar conferences. I guess once you review correctly one they start inviting you for reviewing more. See Youness answer — Carlos Mougan, Jun 28 '22 at 13:38
Ahh, so you have published papers. You only mentioned writing and attempting to write. — Terry Loring, Jun 28 '22 at 13:46
Our group was expected to (and did do) reviews of papers as graduate students. We were U.S., Electrical Engineering, the 1990s and 1980s. I think it is a reasonable request for graduate students. — JosephDoggie, Jun 28 '22 at 20:16
"I have spent already 3 days reviewing and I am not understanding it." Do you normally understand a paper within 3 days and was this all you worked on for those 3 days? Also, while you can't reason your way out of imposter syndrome, consider this. The paper is "very close" to your research topic, so you have probably read similar papers. Compared to those, what makes this one hard to understand? If the exposition is really confusing, this should be mentioned in your review. Other than that, you can comment on the high-level ideas and say the proofs are correct "to the best of your knowledge." — NaturalLogZ, Jun 29 '22 at 01:25
You are not really supposed to push for anything, you should report honestly on the paper. "if the result is true, I find it very interesting", "I have trouble understanding the proof of lemma 4", etc are all very relevant comments. Maybe some other reviewer will have understood lemma 4, or maybe they all have trouble there and it likely means the proof is badly written or wrong. You are not the one making the final decision, you should provide information to the people who do decide. — Marc Glisse, Jun 30 '22 at 07:36
Mate, I wish I had your excelent CV when I was in my second year of PhD. You are doing great. Honestly. I work in ML (since recently) in a maths department, but I don't have math background. There are papers that I know I will just never understand. That is OK, you are not supposed to know everything. Explainable ML is a damn hard topic. — Ander Biguri, Jun 30 '22 at 09:56

score 20 · Answer 1 · answered Jun 28 '22 at 20:47

20

To convert my comments to an answer:

Return the paper to the editors and say that you have hard time reading the paper and suggest they ask a more senior mathematician to evaluate it. Personally, I would never ask a 2nd year PhD student to referee a paper. (However, in the past I did solicit referee reports from PhD students close to completion of their degrees.)

PS. Sometimes, the right answer to a question is "I do not know" or "I cannot do this." Knowing own limitations and being able to acknowledge these, is a sign of mathematical maturity. For all you know, by writing a lukewarm report, you might set up for rejection a great paper, written by an outsider to your research area, bringing nonstandard (for you area) tools and terminology.

answered Jun 28 '22 at 20:47

Moishe Kohan

4,716
3
16
24

9

If PhD students cannot read it, the audience is too narrow. – Anonymous Physicist Jun 28 '22 at 22:06
17

@AnonymousPhysicist You are extremely mistaken. – Moishe Kohan Jun 28 '22 at 22:13
4

@AnonymousPhysicist there is a difference between being able to read a paper and to be able to *peer* review it. – Dikran Marsupial Jun 29 '22 at 07:01
5

@DikranMarsupial But if the reason he can't review it is because he can't even understand it, that seems like a problem. – Barmar Jun 29 '22 at 14:06
@Barmar in this case, it is apparently the detail that is giving the trouble, but it isn't clear exactly why. It may be that it is a case of "proof by intimidation" and the reader is not intended to understand it without unreasonable amounts of effort – Dikran Marsupial Jun 29 '22 at 14:22
1

@DikranMarsupial. Sounds like a poorly written paper. The goal is to communicate, and one way or another it's not doing it well. – Mad Physicist Jun 29 '22 at 19:53
@MadPhysicist The goal is to communicate, but sometimes the intended message is "look at me, aren't I clever?!". Personally, I'd prefer my papers to be useful to somebody ;o) – Dikran Marsupial Jun 29 '22 at 20:00
@DikranMarsupial. If that's the message, it's a crap paper. The content of it may not be crap, but as a paper, it's crap. – Mad Physicist Jun 29 '22 at 20:01
4

@DikranMarsupial: The thing is that neither one of us has any idea what the actual quality of the paper is, etc. Incidentally, many (not all!) ground-breaking papers in math were very hard to read and, superficially, could have been regarded as poorly written. Many mathematicians then spent decades cleaning up the presentation, writing more efficient definitions and proofs. – Moishe Kohan Jun 29 '22 at 20:06
@MadPhysicist Ironically it is often very bright people that do this, and the paper can be valuable in terms of ideas/content, it is just that they waste the time of the reader. The problem is that to be a competent peer reviewer, you need to be able to tell whether it is worth improving the comms. As a reader you can just ignore it as a waste of time. – Dikran Marsupial Jun 29 '22 at 20:06
@MoisheKohan indeed, I was just discussing a limiting case. – Dikran Marsupial Jun 29 '22 at 20:07

Yacine · Accepted Answer · 2022-06-28T22:44:55.413

19

I do not agree with Kohan that a 2 years PhD student cannot review papers at least in my field (AI & ML). PhD students are often assigned by the PC members as sub-reviewers or directly as PC members by the SPC when they have papers in the same (or better) conference level.

Back to the question of the OP; as an author, I NEVER felt that the reviewers have understood my papers to a very high extent (especially when there is a detailed mathematical formulation) and as a reviewer, there are often disagreements between reviewers. In very few cases, I noticed reviewers trying to understand the math behind the papers they review and when they do, they are most probably PhD students because they are keen to read every word in the paper. In contrast, senior researchers have a lot of duties and usually a lot of papers to review so I do not believe they would spend one day per paper. Please note that I am not saying that senior researchers do not review papers seriously but they trust the authors in the mathematical formulation and their reviews are from a higher level perspective (you usually understand what is the role of the equation by understanding its context in the text).

As a reviewer, you are not supposed to understand everything (better if you do) but you need to write the reviews from your perspective and mention to the SPC that you have low confidence in your review. Note also that there is a deliberation in the end and you can adjust your review based on the discussion with the other reviewers.

If you are worried, you can write to the SPC and put him/her in the loop and follow his/her suggestion.

EDIT

After reading the comments, I think I have missed something in the OP's question. The OP mentioned that the paper was assigned to him without giving him the chance to accept or reject the invitation (As far as I understood from the comments). I find this an unusual situation (or the OP did not understand the invitation email) because the common practice in all conferences and journals I know is to send an invitation with the title and abstract, based on which the invitee assesses whether the paper is within his/her scope of expertise. If not, The SPC/PC/Editor expects the invitee to reject the invitation.

edited Jun 28 '22 at 22:44

answered Jun 28 '22 at 12:13

Yacine

5,246
3
21
47

15

For the record: I did not say that a 2nd year PhD student cannot referee a math paper. What I said is that I would never ask a 2nd year PhD student to referee a paper. While it is possible that such student can successfully referee a paper, chances (in my field of math) are not high, as they are typically not ready to evaluate the significance, originality, etc. – Moishe Kohan Jun 28 '22 at 12:20
3

@MoisheKohan I think also that the review practices vary across disciplines. My answer is from the perspective of AI/ML community. – Yacine Jun 28 '22 at 12:25
10

"I do not agree with Kohan that a 2 years PhD student cannot review papers at least in my field (AI & ML)." followed by "I NEVER felt that the reviewers have understood my papers to a very high extent " seems like there may be an association there? This is a big problem in AI/ML - as a rapidly growing field the pool of experienced peer reviewers is much to small for the number of papers submitted and the quality of peer review has gone down rather sharply (resulting in a lot of papers with significant issues being published). – Dikran Marsupial Jun 29 '22 at 06:58
Learning to review papers alongside their supervisor should be part of the training aspect of a PhD, but it is not a good idea to review papers by yourself while still a student. You simply won't have the experience (this is no criticism of the student). – Dikran Marsupial Jun 29 '22 at 07:00
@MoisheKohan Could I have rejected to review the paper? https://twitter.com/zacharylipton/status/1536799053790445569 – Carlos Mougan Jun 29 '22 at 07:04
2

@DikranMarsupial I again do not agree with this statement :) " it is not a good idea to review papers by yourself while still a student". As a counterexample, I am serving as SPC at one of the top 10 conferences in the field and I checked the pool of PCs proposed by all the SPCs. Almost 40% of the proposed PCs are labelled with the position "PhD student". A PhD student is an author in the end and I think he/she -probably- serves better as a reviewer than a senior researcher who has tons of papers to review. – Yacine Jun 29 '22 at 07:50
@DikranMarsupial "This is a big problem in AI/ML - as a rapidly growing field the pool of experienced peer reviewers is much to small for the number of papers submitted". This is an analysis of the fact that PhD students serve as reviewers in some disciplines. I don't think it is the right place to argue whether this analysis is correct or not but what is important for the context of the OP's question is that it is a fact (either we like it or not). – Yacine Jun 29 '22 at 07:56
1

@Younes the PhD students do not have to accept the review request, and they are learning a bad lesson about peer review by accepting if they accept without having the experience to give a competent review. They are seeing peer review how it should not be, rather than how it should, and it would be a bad thing if it just came to be expected. A better approach is to have more unrviewed or lightly reviewed conferences, to take the pressure off the competent peer review pool. As I said before, this is not a criticism of PhD students, experience takes time to acquire for all of us. – Dikran Marsupial Jun 29 '22 at 10:21
1

@DikranMarsupial I couldn't agree more! – Yacine Jun 29 '22 at 11:13
@CarlosMougan: Yes, of course. People decline to referee all the time. As an editor, I typically ask 5 people (on average) before finding somebody who agrees. – Moishe Kohan Jun 29 '22 at 12:24
1

I believe that a typical reviewer of a paper should be able to understand everything in the paper. – einpoklum Jun 30 '22 at 19:34

score 14 · Answer 3 · answered Jun 28 '22 at 13:28

As mentioned in the comments, odds are you are too junior to be reviewing this paper. However, you also mention that they're using nonstandard notation, which is not a good sign. You could return the paper to the editor and say you can't understand it, but there's an easy option: ask your supervisor. They're there to mentor you after all. Your supervisor can help identify if the paper is an obvious reject, and if not, they might be able to point you at resources to understand the paper. It's possible your supervisor will also tell you not to review the paper because you are too junior, but even in this scenario, asking your supervisor is much quicker than asking the editor, so it makes sense to do that first.

score 7 · Answer 4 · answered Jun 29 '22 at 07:58

Publishing papers is not just about getting work out there, it is also about making other people understand the work done. As a 2nd year PhD, you are already part of the target audience of papers in your field.

As a reviewer, it is perfectly fine to come to the conclusion that a paper is not properly presented. Do not hide such a conclusion just because of the conclusion itself.

If your general impression is good, then a weak accept may be appropriate. If you are basically guessing the content, then a (weak) reject is no shame either.

My impostor syndrome when seeing overly mathy papers.

This is really where you should put your focus on. Is the paper hard to understand because you generally struggle with mathy papers? Is the paper hard to understand compared to the usual mathy papers?

In a proper review, you can rate you degree of confidence. If you generally struggle with such papers, then a low confidence score is appropriate. This allows other, more thorough reviews to take precedence.

Their math formulation is not the traditional one and they do naming and formulation variations. Even though they can be properly defined.

This seems like a clear shortcoming of the paper. Using obscure formulations – be it for math or prose – can hide veritable errors or blunders. At the very least, it makes it needlessly difficult to understand and/or reproduce the findings – one of the key goals of publications.

If you are feeling unsure about this point, check the paper's references. Do those use the same formulations? Is there a consistent formulation for key points that the reviewed paper deviates from?

Over relies on the appendix to understand the content of the main body of the paper.

This depends strongly on the field and publication formats. In some it is perfectly fine to use the appendix for background and explanations that is already known to scientists in the field. In others an appendix is more of a technical addition to thoroughly reproduce a paper.

If you are unsure about this, note it as a comment to the editor. There may be formal rules about the appendix that have not been communicated to you.

The paper content is very dense.

This ties in with some of the above points. Is the paper very dense compared to usual papers? Does the denseness make the paper needlessly hard to understand?

Try and identify specific points where content seems either superfluous or too short. It is fine for a review to suggest that some parts should be dropped in favour of expanding others. Take into account the appendix as well; it might be acceptable in the field to have a dense paper and supply lengthier explanations extra.

score 3 · Answer 5 · answered Jun 28 '22 at 12:06

I would suggest giving the paper a general fairly positive review, or a lukewarm review, based solely on the high level introduction (if that's what you feel), while providing the lowest confidence degree (if that metric is available), and writing to the editor in the "note to editor (wouldn't be revealed to author)" that you understood the high level but not the details or technique and couldn't verify the proofs/arguments.

If you don't understand anything in the introduction, I would write that as far as you see you couldn't understand the merit of the paper from the current presentation (still specifying low confidence degree), and explaining that this is due to either inadequate exposition or your own misunderstanding (e.g., "I couldn't understand the overall merit...").

Big_Al_Tx · Answer 6 · 2022-07-01T02:49:44.027

Well, I have a Master's Degree (not a PhD) and I've reviewed dozens of papers in my career, and I think I was more than sufficiently qualified to review them (even when some were written by PhD's). I don't believe the degree (or lack thereof) is the real problem -- a 2nd year PhD candidate should have more than enough academic knowledge to review technical papers. If the OP has not spent any time outside of school (i.e., from BS & MS directly to PhD), then it may be more a matter of lack of experience in the "real world" where they could have been exposed to more work being done in that discipline. Please don't take that as a criticism; it's more of a potential explanation of why they may be having some difficulty.

As far as reviewing papers is concerned, I use these general "rules":

A paper to be presented at a conference or published in a journal is a way to present the results of some research or study to the world so others can benefit from that knowledge. The key is: COMMUNICATING the work done and the conclusions drawn from it. The subject (problem statement, proposed approach, assumptions, etc.) need to be communicated clearly enough for a practitioner in the field to understand, even when they aren't "experts". In this case, it sounds like that paper isn't doing well on this point.
There should be enough explanation in the paper to allow a practitioner in the field to replicate the analysis described (although the data itself may not be accessible), and should include a sufficient level of detail on how that analysis was done. Obvious and common methods (like how to do ANOVA, for example) shouldn't be needed, but constantly referring to external references in the Appendix without providing specifics to more involved methods/tools does make it difficult for the reader to follow what the author is describing. It also doesn't allow the reviewer to determine if the author actually understands those methods and knows how to apply them correctly, specifically in the context of the paper.
Also, if the author is not following standard notation or methods used by other practitioners in the field, then it undermines the 1st bullet (communicating clearly) as well.

There is nothing wrong with saying that major portions of a paper are unclear and confusing -- I've met many people who were clearly technical experts in their field, but were extremely poor communicators. That's made even worse if the author is writing a paper in English, when English is actually their second language. I know they can't help that themselves, but it is up to them to seek out a native English-speaking colleague to proofread and provide feedback before submitting it for a formal review. If that is the kind of review you're doing, then the feedback you provide is that much more important.

If it's really beyond your level of knowledge, there's nothing wrong with saying that you're not sufficiently qualified to do the review.

But DON'T just approve it to get it off your desk (there's far too many "junk" papers out there that should never have been published because some reviewers were a little lazy).

At the same time, rejecting something just because you don't understand it may deprive others from being able to benefit from that research.

That's just my opinion, but I hope my perspective helps you decide what to do.

Inability to review a conference paper

6 Answers6