19

I know, this is a bit of a broad question. But looking at the developments and having experienced a reasonable amount of pain with established publishers and their either complex, slow, or restrictive processes/production stages, I'm tempted to ask it here and look what others think or know about that.

Post-publication peer review has been discussed quite a while ago here and here, it's been promoted and practised on platforms such as Publons and PubPeer.

But is this idea, launched many years ago to complement "classical" peer review, really successful? Are established processes/practises being replaced/improved?

Note for clarification: I envisage something like a combination of classical pre- and an optional post-publication peer review strongly based on open self-archival platforms such as arXiv. Nothing on the academic side would change, editors and reviewers would move over to such platforms to continue their work just like before, largely unpaid. Even cult-like reputation-building, gate-keeping, etc. is still possible even if not desirable. But in domains such as CS, physics, or math, where production is largely done by authors anyway, costly, error-prone, after-acceptance production stages would be replaced by just self-archival and EiC-driven quality labelling systems as they have been used for years at top-tier conferences to certify such things as reproducibility. I think, it's time for a change. (I wonder whether this question is more something for meta.)

Further note: After the many useful comments and answers, I'm extending or rephrasing my question to: Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

mfg
  • 1,453
  • 15
  • 22
  • 1
    Editors and reviewers are unpaid, but the editor's secretary, who keeps track of who all the reviewers are and whether they need a reminder or not and tracks down the reviewer who changed jobs but forgot to tell the editor their new e-mail address (software can help with this but can't do all of it) has to be paid. – Alexander Woo May 06 '21 at 21:54
  • One needs a mechanism to encourage such reviews to be collected. For example, all referees/reviewers for awards and promotions could be requested to review at least one paper in order for their letter to carry additional weight. – Kapil May 07 '21 at 04:46
  • 1
    We need a robust reddit-style comment system on the Arxiv so that people can discuss papers easily and openly. The comments will be very enlightening. The comment system should support upvoting and downvoting comments and should support mathjax. This needs to exist. – littleO May 08 '21 at 02:56
  • @littleO aren't conferences and coffee breaks existing to do that? plus, having a written trace or a permanent record actually prevents open and easy discussion. Academia is power unbalance, retaliation fear is too high, try to naively hit a weak point of one of the best recognized professor in your field as a PhD and you will see how long it will take you to recover if he/she is a bad-temper person... – EarlGrey Dec 07 '21 at 09:29

9 Answers9

31

Most papers that are published are uninteresting.

The median paper has less than one reader not counting the authors and the reviewers.

The current reviewing system relies on a network of responsibilities that are independent of the author. The editors have a responsibility to the community as a whole to get papers reviewed properly, and the community gives them esteem for performing this duty. The reviewers have a responsibility to the editor to actually do reviews, and the editor is usually some influential person (and in any case will probably at some point be the editor for one of the reviewer's papers).

Suppose I have written one of my usual mediocre papers that is among the 70% (my estimate) of papers that are never read. How would post-publication peer review be organized for it? If I have to find reviewers myself, then everyone is going to say 'No' or do a bad job, because no one feels any need to impress me and it's not a very interesting paper to read. Moreover, the best reviewers for my paper are people with a little higher status than me, and they would definitely say 'No' to me, but they might say 'Yes' to an editor. (Keep in mind that, as the system is set up, some top people in a field do far more reviewing than they write papers, while many people write one write-only paper every several years (or grad students who stop doing research write one paper from their dissertation) and never review.)

Alexander Woo
  • 24,741
  • 3
  • 59
  • 91
  • 5
    Yup! Papers are awful and refereeing is the worst. There is one other motivating factor here that you don’t mention, I referee young mathematicians work (even if I don’t feel obligated to the editor) because I want them to get jobs. But that mostly applies just to postdocs. – Noah Snyder May 06 '21 at 15:57
  • 3
    Note that post-publication (or non-publication) peer review happens pretty well for interesting papers that people want to read. Perelman’s work went through a very extensive process to verify the work and expand on the details despite never being published. Conversely, everyone knows where the hole is in Mochizuki’s work from outside readers, even though it’s published and the referees didn’t do their job. – Noah Snyder May 06 '21 at 16:03
  • 14
    Actually, it isn't that most papers are uninteresting. It is that the audience for extremely specialized work is limited in many fields. – Buffy May 06 '21 at 21:53
  • True, and it's often impossible to read/follow all relevant work even in a very special field. – mfg May 06 '21 at 22:01
  • In practice there is no accountability for those "responsibilities." – Anonymous Physicist May 06 '21 at 23:25
  • 7
    Do you have a source for The median paper has less than one reader not counting the authors and the reviewers? I can ask a separate question on this if you consider it off-topic for this one. – Allure May 07 '21 at 06:10
  • @Allure I don't have a source but it seems completely reasonable to me for papers in pure maths. It might also depend on what you mean by 'read'. I published half a dozen papers. For everyone of them I would say 5 to 10 people read the abstract and downloaded the pdf, maybe skimmed it for 5 minutes or so. Two of them where cited, in one and two other papers, all my other papers are uncited. I would only count the latter as 'reading' which gives the claim in the answer. – quarague May 07 '21 at 09:00
  • 1
    @Allure - No I don't think anyone has done a study on this, in part because 'read' is hard to define. If, by 'read', you mean 'read in as much detail as I would if I were a reviewer', then I more or less only read the papers I agree to review (or the ones I write) and certainly read fewer papers (not counting reviewing) than I write. (And, by this definition of read, I do not come close to reading everything I cite.) – Alexander Woo May 07 '21 at 10:52
  • Why is it necessary or desirable to peer review all papers? Let us not waste precious time with bad papers. – Sylvain Ribault May 07 '21 at 20:57
  • 2
    @SylvainRibault: Because research is not the only purpose of research. I know someone with a PhD who got their bachelor's degree from Alma College. There, the professors teach 360 classroom hours or more a year, leaving little time for research. Alma College and the mathematical community would like to know that the professors there have a good sense of what mathematical research is so that we can trust the recommendation letters they write (and for other purposes). So we want to review their occasional papers even though they don't seriously advance mathematics. – Alexander Woo May 07 '21 at 21:14
16

In a sense, it has taken off; you just need to change your definitions: in mathematics, most papers get peer reviewed after they have appeared on the Arxiv and are available to the public.

Results then appear in prestigious journals, so they are ultimately trusted and used to evaluate researchers and compare academic egos, only after they have been carefully been reviewed; but that seems working as intended and I wouldn't want it to be different.

Federico Poloni
  • 46,039
  • 18
  • 129
  • 194
  • 1
    Thx Federico. Totally agreed, pre-publication peer reviews are a must also IMHO. But I was more (perhaps in an unclear way) hinting to the idea that the often error prone production stage could be replaced by arXiv or similar throughout the whole process. Big conferences have years ago started to label papers with "evaluated" or "reproduced" etc. Why can't we use such labels for arXiv? And "label management" would then be done by EiCs by help of reviewers if you like. – mfg May 06 '21 at 13:29
  • Review/editorial work is largely voluntary anyway. I wouldn't mind to do it equally for arXiv like for any other publisher, given I am willing to put trust in the corresponding EiC – mfg May 06 '21 at 13:34
  • The journal-ref field on Arxiv already serves a similar purpose, in the end. – Federico Poloni May 06 '21 at 13:38
  • I see, is there a particular process (on the arXiv side) associated with journal-ref that would avoid the need for going through the production stage of a publisher? Hmm, maybe we are talking of two different things here. – mfg May 06 '21 at 13:42
  • I see what you are looking for now. Many publishers are now reasonably quick to give you a DOI and put the paper online "preliminarly". And there some Arxiv "overlay journals" where the paper is published on Arxiv only. But I understand none of these answer your question exactly. – Federico Poloni May 06 '21 at 13:47
  • No, I think you are going in the right direction. The question is: can we be progressive and just spare the production step (which is at least in my field error prone and very costly). I might edit my question to make it clearer. Will do that at night. – mfg May 06 '21 at 13:49
  • With the edit I suppose this is the right answer, what you envision is already exactly the mathemaical and physical sciences work, except for an unimportant part at the end where eventually someone prints it up (but people still read papers primarily on the arxiv). I don't understand why you care so much about eliminating this last step which doesn't really matter anyway. (If your goal is to save money then the answer is that for various reasons biomed does things differently, and if you have a "big deal" you save nothing by cancelling some subscriptions.) – Noah Snyder May 06 '21 at 20:15
  • @Noah: All good points. Then let's copy math/phy practices. So, I want to save the last step because it postpones publications, proof preparation is error prone, proof correction cycles can take a lot of time, subscriptions or APCs cost the system a sh*d of money, publishers are working with old editorial systems, restrictive LaTeX settings, ... I could go on like this. But sure, there will be cons as well. I just don't see them right now. – mfg May 06 '21 at 20:35
  • I find that the proof part, though error-prone, isn’t that annoying because I mostly ignore it. The delays don’t matter because no one is reading the journal version anyway and I can already list a paper as accepted on my CV even if it’s not published. The cost part matters, but there the answer is all about the “big deal” and that med schools are really rich and don’t care about spending crazy amounts of money. Math journals aren’t really costing universities money since they have to subscribe for biomed anyway. – Noah Snyder May 06 '21 at 21:02
  • @Noah: Thanks, these are interesting insights into a domain unfamiliar to me. In CS, it's quite different. Subscriptions are being replaced by gold OA, but that's expensive too. It doesn't change much for us. Ok, so it depends, I guess. – mfg May 06 '21 at 21:59
  • Yes. There is also a somewhat related math comment in Astronomy SE question about arXiv. – uhoh May 08 '21 at 09:52
7

In a sense, there is post-publication review: It just doesn't happen on websites, but in subsequent papers written by others. And the opinions of others are also recorded: A citation is like a "Like" on Facebook: The citing authors thought the cited paper offered some useful background to readers of their authors.

In other words, papers that are highly cited are, in some sense, positively post-publication reviewed. The converse is also true.

Wolfgang Bangerth
  • 94,697
  • 7
  • 201
  • 338
  • Very true! :) Good point! :) – paul garrett May 06 '21 at 21:20
  • 1
    This is not correct because people also cite papers to show that they are wrong. – Anonymous Physicist May 06 '21 at 23:28
  • 2
    @AnonymousPhysicist You're nitpicking. Which fraction of citations overall does this apply to, and which fraction of highly cited papers are highly cited because others are complaining about them? – Wolfgang Bangerth May 07 '21 at 05:50
  • 1

    "people also cite papers to show that they are wrong" doesn't matter - it's still a form of review. Or, what should "post-publication review" do about bad papers - delete them completely? What should be the threshold for doing so?

     – Lou Knee May 07 '21 at 21:50
6

In a sense, post publication peer review has always been in place. Prior to publication peer review has the purpose of improving a paper before it appears (or at least prior to its formal publication). This is normally a private matter between a journal or conference and the authors.

Post publication, however, peer review has a different purpose and can no longe improve that paper. But anyone reading a paper can comment on it, make corrections to it and publish extensions or corrections. This has always been the case. Once a paper becomes visible it is open to comment by any "reviewer". This has, of course, let to some mighty intellectual battles, such as Leibniz v. Newton. If memory serves, Einstein's early papers weren't universally accepted but created some turmoil among the great names of the day. They generated a lot of comment, but also a lot of additional work attempting to refute or verify or extend them.

Buffy
  • 363,966
  • 84
  • 956
  • 1,406
  • Thanks, Buffy. That's why I'm trying to suggest a combination, just without the production stage via a publisher. Because, talking mostly of CS/math, I believe most of the typesetting is done by authors anyway. So, established scientific quality assurance wouldn't change (at all). – mfg May 06 '21 at 13:38
  • 1
    No, you are wrong. Pre publication peer review isn't about typesetting. It is about correctness and impact as well as assuring that he author properly situates the paper within the literature (citations and such). For print journals, I also suspect that typesetting is done by professionals. – Buffy May 06 '21 at 13:41
  • Sorry, I was making myself not clear enough in the question: Assume we have the scenario thinking - writing - submitting - pre-pub peer review - revision - production - publication - post-pub peer review - reading - citing - etc. I wonder whether production + publication can just be omitted by staying on arXiv? – mfg May 06 '21 at 13:45
  • 1
    Then no paper can be trusted to have been vetted by experts. I can publish on my own website, actually. If you want to trust me you have to do it on your own. – Buffy May 06 '21 at 13:49
  • No, that's not what I meant. Just shifting the voluntary EiC/reviewer work from publisher to e.g. arXiv. Nothing essential on the academic side would change. – mfg May 06 '21 at 13:51
  • 1
    But the publisher need their own assurance of quality and uses reviewers they trust. – Buffy May 06 '21 at 14:49
  • 2
    One could perhaps consider citations as post-publication review. – Jon Custer May 06 '21 at 15:44
  • 6
    @Mario: You seem to be missing that at least 95% of the point of "publication" is just being able to write "accepted by J. Prestigious Results" on your CV. Everything around copyediting, production, printing, etc. is basically unimportant. – Noah Snyder May 06 '21 at 16:55
  • @Buffy: I'm referring to a system where there is no publisher expect for self-archival. But there are editors and reviewers as is, and they carry through peer review at certain stages (with one stage corresponding to the "publication" stage, called pre-pub, post-pub, or else). I've modified the explanation in the question to make this clearer. – mfg May 06 '21 at 19:59
  • @Noah: That is a valid point. In that pessimistic case, my question is sort of useless. But I'm still not that pessimistic. – mfg May 06 '21 at 20:13
  • 1
    If everyone puts out whatever they think of, how can we sift through the whole pile and pick out what is actually quality? It's hard enough as is to find the good publications, think about how much worse it would be if there was 10x that much, completely uncurated. – Wolfgang Bangerth May 06 '21 at 21:10
  • @Wolfgang: Thx for your thoughts. I should note that I've never said anything about abandoning pre-publication peer review (actually I try to make the opposite point). I tried to make this clear in most of my comments and even in the original version of my question, I mentioned to keep pre-publication peer review. Peer review must remain THE quality assurance instrument. Absolutely agreed. – mfg May 06 '21 at 21:48
6

Because post-publication peer reviews are not currently recognized as useful contributions, so why do them when I could do something that would help me get a job?

Azor Ahai -him-
  • 30,111
  • 9
  • 88
  • 115
6

As Tim Gowers wrote in 2013 in the context of an experimental peer review website:

[...] It is easy to come up with ideas for websites where people can review papers, complete with clever protocols for how the reviewing should take place, whether it is open, reward systems, etc. etc. It’s much less easy to persuade people to use the sites that are created as a result: what is going to persuade them to make the effort, when there’s only rather a small chance that the site will become in any sense “official”?

"Becoming official" is the problem here: the publishing system's main function is no longer scientific communication, no longer improving and evaluating papers. The system has been coopted by administrators and funders, in order to manage researchers' careers via bibliometric indicators. Its continued dominance makes it difficult for alternative modes of peer review to emerge, since any new initiative must compete with an entrenched, official system.

Sylvain Ribault
  • 2,132
  • 7
  • 11
  • 1
    Thx, @Sylvain. Let's leave aside post-peer review and stick to the best we have: pre-pub peer review. I believe in certain fields, there are reasonably well established platforms, such as arXiv or the like, that could (AFAIS) be extended to perform high-quality classical peer review and curation, just without the money-eating/encrusted structures of traditional publishers. – mfg May 07 '21 at 10:23
  • 1
    Yes, arXiv could do it, but it is not doing it and may never do it. ArXiv could have done it from the start, but did not want to scare publishers. Now it is probably too late: after "becoming official", these platforms do not evolve much. – Sylvain Ribault May 07 '21 at 20:55
  • When my uni made me start using the institutional Figshare a few years back, I'm sure they still had a "Comments" feature; no sign of that recently. So moving away, rather than towards. :( – Lou Knee May 07 '21 at 21:45
  • IMHO, the feature arXiv would need is to allow to label archived preprints as e.g. archived, peer-reviewed, evaluated, accepted, produced=published, etc. and make sure that those people currently voluntarily working as EiCs for journals are the only ones allowed to assign these labels exactly following their current editorial practices. I think there is a weak form of such a feature there (maybe called endorsement, don't know), but I don't really understand why it hasn't taken off. – mfg May 08 '21 at 09:36
5

As in the other answers, the premise of the question is a bit inaccurate.

For work that turns out to be important, people look at it critically (with interest) long after it has supposedly been "refereed" and "published" (in the sense of being endorsed by a journal). And will look at it critically (with interest) as soon as it's available (e.g., online), whether or not it's submitted to a journal.

And, as has been true for 20+ years, there is the internet. Public availability of documents is no longer monopolized by "publishers" in the traditional sense. Of course, some people are less shy than others about putting online dubious stuff.

If one wants to be confident of the correctness of a journal document, one should probably check it in detail oneself, unless it is a very high profile case. In recent years, the guidance I've gotten as referee is that it is not my duty to check for correctness! And, well, in cases where no one cares much whether the conclusion is correct or not... ?!?

In math, in the U.S., we are still in the situation that, in most places, the only official way to score career/status/tenure/promotion points is by publication in refereed journals. People scrambling for tenure are not going to throw away their professional currency. And, in my experience, many people who've succeeded in this system are loathe to "make it easier" for the next generation. (I myself have recovered from any affection for artificial suffering... :)

paul garrett
  • 88,477
  • 10
  • 180
  • 343
  • Thx for these valuable insights. I am aware that step changes like I suggest won't happen without loss. But still, I think, the way how established publishers work is prehistoric. There are platforms like PLOS or f1000research which however are just not AFAIK really established in my domain. – mfg May 06 '21 at 20:43
  • This does not answer the question because it's not peer review if the outcome is not public. – Anonymous Physicist May 06 '21 at 23:26
  • In recent years, the guidance I've gotten as referee is that it is not my duty to check for correctness! Sounds surprising (and ridiculous, and sad). Could you elaborate a bit? – Richard Hardy May 07 '21 at 17:59
  • 3
    @RichardHardy, yes, it is surprising, since we might think exactly the opposite. Nevertheless, in the last 10-20 years, quite a few substantial journals' official statements of what the referee's task is have said that it is not the referee's job to determine correctness. – paul garrett May 07 '21 at 18:08
5

I'm taking the definition as provided in OP:

Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

The first part has already "taken off" and is standard in my field (physics). The latter part, post-publication commenting, has not. The reason is, why should it have taken of? For something to supplant the original, it needs to be better. Post-publication commenting is not better for several reasons:

  • What would be the point? Pre-publication review can conceivably lead to the paper being improved. Post-publication review cannot.
  • What are you going to say? You can't provide suggestions for improvement because the paper is already published. You could say "This paper is great, I enjoyed reading it" which is nice to read but doesn't lead to anything. Meanwhile "This paper is bad, it should never have been accepted in the first place" is painful for the authors, especially if the criticism is public, and especially if the criticism is anonymous. Example I am aware of.
  • What would one gain from making the post-publication comment? Think about why people do peer review in the first place, and compare that to whether people should do post-publication commenting. Many of the reasons simply do not transfer: the results are already published so you don't get to read new exciting stuff before it's published, these things aren't organized by journals so you gain nothing from that front, and you can't say you're an acknowledged expert because odds are anyone who wants to submit a comment can do so.
  • Finally, it's not a reciprocal relationship. Traditional peer review is reciprocal - you peer review for others, and in return they will peer review for you. The same does not apply for post-publication commenting.

Everyone knows the traditional peer review model is flawed, but coming up with a better alternative is pretty damn hard. Post-publication commenting is an example. You might be interested in others - off the top of my head there is open peer review, where the identity of reviewers is public, and the accept-then-review model where the journal accepts every paper before putting them up for review.

Allure
  • 127,528
  • 50
  • 325
  • 493
  • 3
    But even if the paper can't be improved in response to the comments, the comments themselves could constitute valuable additions/improvements to the paper. Just like comments here on SE often add value to an answer even if they aren't edited into the answer. There's only so much that pre-publication peer review can do, the 100 post-publication readers will always find some unclear bits that the two per-publication reviewers found perfectly clear. If the authors could easily and publicly respond to requests for clarifications, most papers would get much easier to digest. – TooTea May 07 '21 at 12:03
-1

Post publication peer review has a model that has proven to work, in both principle and in practice: Stack Exchange. The "research question" is generally posed by someone other than the author; there are multiple short "articles" answering the research question; peer review consists of comments and, in many cases, other articles under the same question; and, ultimately, a quantitative rating is given to the quality of the question and most of the articles, with the original questioner getting a binary (checkmark) say as the "editor."

Imagine if, for example, someone posted a question on a (relevant subject) exchange to the effect of: "Is the linked article good?" Then, assuming it didn't get taken down for being off-topic or overly vague, the resulting exchange and ratings would constitute PPPR by almost any definition. (Wikipedia would be another example, but it doesn't allow original research. You might say that little on here consists of original research, but that's really only true for the quantitative, empirical sciences, and even then a lot of published journal articles just aggregate known information.)

Most of the reasons for PPPR's failure to launch, as given in other answers, are refuted by Stack Exchange. This community consists of thousands of people, including "real" experts, who are willing to read and respond to what frequently are indisputably boring or obvious questions, for free, in an unofficial forum. Some even perform basic copyediting! True, the questions and answers, individually, are shorter than an article. But Wikipedia shows that people are also willing to read and contribute to much longer articles. And sure, there's little incentive for authors to submit their work to an "unofficial" site, but as the above example shows, you don't need the author to submit it themselves.

So, either Stack Exchange has uniquely captured lightning in a bottle (in which case it should establish its own PPPR forum) or PPPR sites have failed to copy a remarkably effective format. And that's the answer to your question.

virtuolie
  • 101
  • 1