PubPeer and ongoing evaluation of past published papers

In my capacity as Editor-In-Chief of The American Naturalist, I sometimes receive emails criticizing already-published papers in this journal.  The criticisms sometimes come in the form of anonymous emails to me. Other times they come from known emails and contain lengthy attachments with detailed criticisms. Increasingly, however, these criticisms are posted publicly on PubPeer, often by anonymous individuals. I want to take a moment to comment on this trend, from my perspective. Note that the following is not formal journal policy, and other Editors  (current and future) may disagree.

The individuals posting anonymously on PubPeer are typically genuinely concerned members of the scientific community who have identified errors, or strongly suspect errors exist, in the scientific record. Their criticisms should be addressed for their technical content. Left unanswered, a cloud hangs over the paper(s) in question.

That said, there is some question about the proper procedure for answering these criticisms. Yes, PubPeer itself leaves room for comments (interestingly, journal editors like myself must pay money to reply to comments, even if to acknowledge them and state we are evaluating the issue). But, this process bypasses the journal that publishes the paper, and bypasses the normal scientific tradition of external review by experts in the field chosen by the journal editor for their knowledge and hopefully objectivity. For this reason, I want to really encourage people with substantial concerns about a paper (e.g., which may appreciably alter the results and conclusions), to submit formal "Comments" (different journals call these different things) to the journal. These Comment manuscripts get reviewed (including by the original authors), and the original authors may write a Reply. Based on the reviews and comment and reply the Editor decides whether to (1) publish just the comment, (2) publish the comment and reply, (3) request a correction or retraction, or (4) let matters lie because the criticism does not have solid ground to stand on, is deemed to be debatable, or (per COPE guidelines) if the issue has no material impact on the conclusions of the paper. 

I know that the Comment approach is onerous in several ways. First, it takes time to prepare a formal manuscript formatted submission. I know of at least one instance (concerning a putative error in an equation in a theory paper) where the critic took time to write a pages-long missive, but said s/he didn't want to take the time to actually turn it into a Comment.  Second, the Comment is authored and so the critic identifies him or herself to possible reprisal. Recent activity by lawyers of one author (letters, FOIA requests) illustrates the point that critics can be subjected to legal and other harassment, which is a stressful penalty for free-speech in pursuit of ensuring scientific accuracy. {Note; anonymously authored Comments are not traditional, but might possibly be considered}. Third, Comments go through review which takes time, leaving the matter behind the scenes in the meantime. Here is where things get tricky: if a criticism proves to be groundless, a PubPeer criticism can damage an author's reputation needlessly and inappropriately. On the other hand, if a criticism has merit, we'd like it to be public knowledge immediately to correct the scientific record as soon as a problem is identified. These are conflicting needs. PubPeer is on the side of rapid dissemination, and reviewing Comments behind the scenes can err on the side of perpetuating errors too long (and can also open journals to criticisms of foot-dragging and cover-ups). While perhaps such cover-ups exist, every case I am directly aware of I have been satisfied that journals are genuine in their desire for ensuring quality science, but must exercise caution in leaping to conclusions without due process (which occurs behind the scenes).

So, as a journal, our official stance is that we prefer to receive submitted Comments. This follows the scientific tradition of obtaining third-party review, and deliberate evaluation of criticisms. But, scientific traditions are fluid and we are in an era of increasing speed and openness:  Preprint servers, open peer review, open data, et cetera. We therefore also recognize that PubPeer is an active tool in science conversations. The criticisms posted there can be valid identification of genuine problems that need to be evaluated formally and corrected. If valid well-justified and substantial concerns exist and are published on PubPeer, then the affected journal should respond.

There are caveats and concerns about PubPeer. Most importantly, it is important that these not be used as a mechanism for pursuing personal vendettas. Excessive targeting of an author with multiple minor complaints can constitute a kind of harassment, and may be viewed as such by University Equity officers or equivalent. The anonymous nature of many PubPeer comments makes it easier for impacted authors to feel like (and, argue that) they are the target of personal vendettas and harassment. Second, the existence of PubPeer comments can cast a long shadow over a paper whether the comments are profound or minor.  This shadow can affect an author's career prospects (fellowship applications, job applications, etc) even before the matter is resolved and judged to be valid or not. The result can be inappropriate damage to an innocent authors' career, which in turn may have grave consequences for mental health. Third, there is an established mechanism for voicing complaints about papers in science:  contact the author to request clarification, or contact the Editor, or submit a Comment. I realize that many complainants are nervous about revealing their identity, and these three steps I list make this harder. One can create new email accounts to anonymously contact an author, or Editor. But bear in mind that although scientific publication is in no way bound to follow legal court-style procedures, there's something to be said for the legal tradition that accused have a chance to face their accuser. Science publication is not a court of law; but it is worth considering whether we as a community want to respect that kind of tradition. That said, whistleblowers sometimes hide their identity for good reason (e.g., not trusting that whistleblower laws offer sufficient protection). With this in mind, it might be worth considering whether journals might publish anonymizied Comments. That runs counter to the growing trend towards open peer review, but can protect vulnerable individuals when their complaint is valid and they have compelling grounds to be concerned about how their career is impacted.

How should journals respond to PubPeer comments? This is something of a wild-west issue that I expect COPE (the committee on publication ethics) is grappling with to provide guidelines. First, the PubPeer comments need to be brought to the Editor's attention. Don't assume that Editors take time to search PubPeer regularly; at least I don't. Then, the journal begins an evaluation process that may include (1) forming a committee to evaluate the papers and the citicisms, and/or (2) contacting the author(s) for a response. These take time, indeed can take many months to do properly with due care for the details and reaching a scientifically defensible and just decision. So, please be patient. Sometimes there are delays due to valid extensions in recognition of health concerns, parental leave, etc, that need to be appropriately accommodated even in pursuit of ensuring scientific truth. Also, do not expect any formal journal response on the PubPeer website; they charge Editors money to make any statement as Editor (that's how they pay their bills, at least in part).

Obviously if there is compelling evidence of malpractice by authors, the investigation should ultimately result in retraction (e.g., a Comment alone ceases to be sufficient). But, some cases where a critic may reasonably suspect data have been fudged, may appear to others to be less clear-cut. A journal's job is to carefully evaluate the opposing claims and the weight of evidence. The result may end up being inaction where the journal's editor or their appointed committees are unconvinced by the criticisms. It is my belief that, if a criticism proves to be invalid, then its public airing on PubPeer can do more harm than good. Which is why I prefer such processes be handled via a formal evaluation and review.

To summarize, I believe that PubPeer has appealing features, but has the potential to do harm to innocent authors, or to be used for personal attacks. Our goal should be to promote its strengths while mitigating the risks.  As editor I therefore prefer that critiques begin with direct communication with an author, and that journals be notified via submission of a Comment when those direct communications do not yield a satisfactory answer (or, go unanswered). In this way, concerns can be evaluated and questions can be answered, through a formal review process that might resolve the issue without exposing the author to public ridicule that might ultimately prove to be baseless.  That said, I understand the motive for using PubPeer (anonymity, speed, openness) and will respond when made aware of PubPeer critiques that appear to have scientific merit.

I am aware of PubPeer criticisms targeting multiple papers of some authors. These cases take extra time to evaluate because they entail more work: committee members need to find time to read many papers, delve into their data, and authors need to respond to more comments. Please be patient. To the broader community, a journal may appear to be ignoring criticisms; but this may simply be the result of a careful and incomplete process. In some cases we also are awaiting COPE recommendations for how to proceed in cases that reveal new kinds of publication-ethics problems.

In particular, we are aware of cases where published papers lack complete data repositories (e.g., on Dryad). A present, we are awaiting guidance from the Committee on Publication Ethics on how to proceed in such cases.  This is a new enough situation, that the recommendations are not yet clearly articulated as a scientific community. As an interim policy, our first step is to notify authors and give them a deadline for completing their files to our satisfaction. Such deadlines may take some time, for instance if hard drive forensics are needed, or if an individual is on parental or health leave. If the data files cannot be completed by the deadline, our present plan is to issue Editorial Expressions of Concern. Authors with such EoCs will have any future papers scrutinized far more closely for complete data. Authors would do well to avoid such situations by uploading complete Dryad (or equivalent) archives following best practices recommended by Dryad for thorough meta-data, ReadMe files, etc.

 As an institution The American Naturalist is dedicated to publishing rigorous, well-supported research, and that rigor takes time. Every other Editor-In-Chief of other journals whom I have interacted with extensively shares that commitment. We will never sweep valid criticisms under the rug to defend our reputation, because that reputation comes not from having papers, but from the quality of the papers that we disseminate. But, sometimes the deliberate process of internal review may appear to the outside to be inaction; often nothing could be further from the truth.

Ultimately, it is journals that publish science (for now). And thus it is journals' job to correct errors in the science that they previously published. It follows that for any serious scientific error, it rests in journals' hands to evaluate the error and take suitable corrective steps.  Of course, if journals truly fail to meet that obligation, that's a problem. And PubPeer represents an alternative, a safety net for criticisms, and a way of shaming journals into action. But - I'll say this again - sometimes what looks like inaction from outside is either (1) a slow deliberative process to ensure complete and fair evaluation, or (2) a completed evaluation that found nothing to move forward with. So, please give journals some time to do what they need to do. 

Dan Bolnick

Addendum:

A Twitter comment pointed out that PubPeer enables author responses to critiques, creating online dialogues that may resolve and clarify and issue. I appreciate and value that. The poster (Boris Barbour) also notes that most PubPeer comments are not graced with author defenses or acknowledgement. If correct (no statistics were offered in support), that is troubling. And in my view that again points to the value of journals as the enforcers whose job is to follow through and either require a response or else publish a retraction or Expression of Concern or Comment. Authors can ignore PubPeer comments because... because they can, there's no immediate penalty for engaging besides the court of public opinion.