September 8, 2020

Guidelines for data on Dryad

 The American Naturalist has begun the process of creating a new editorial board position: 'Data Editors'. Their job will be to help facilitate and check compliance with the journal's open-data policies. We require publication of raw data and metadata on Dryad or equivalent public repositories, and we recommend including code to reproduce analyses. Exceptions can be made, to post the data but embargo public access for a set amount of time to allow authors to publish related papers, but we rarely get these requests.

It has come to our attention recently that compliance with our data policies is ...  not quite as effective as we'd like. This mostly comes down to missing material in the posted data, or unclear metadata. To help authors do a better job of meeting our data publication expectations, I began a committee composed of Bob Montgomerie, Paulinha Lemos, and Rob Knell.  They have produced the following Data Archiving Checklist, listing best-practices which authors may find useful in preparing materials for Dryad or other data archives. You may also find the DRYAD best practices list useful.

Checklist for preparing data to upload to DRYAD or other repository

 

*  assemble all of your data files used to prepare your paper

*  ensure that every observation is a row and every variable is a column

*  if you analyzed means rather than the raw data, also supply a data file with the raw data from which those means were calculated

*  from each file remove variables not analyzed

*  identify each variable (column names) with a short name (no spaces or symbols), preferably <10 characters long. Use an underline (e.g. wing_length) or camel case (e.g., WingLength) to distinguish words if you think that is needed. See the Google R style guide (https://google.github.io/styleguide/Rguide.html) and the Tidyverse style guide (https://style.tidyverse.org/syntax.html#object-names) for more information

*  prepare a README file that lists all of your data and code files with a brief description of the file and a list of all variable names and an explanation of each variable so that someone else could understand what that variable means (including units). See Dryad suggestions here.

*  save the README file as a text (.txt) file and all of the data files as comma-separated variable (.csv) files

*  if your data are in EXCEL spreadsheets you are welcome to submit those as well (to indicate colour coding and provide additional information (formulas etc) but each worksheet of data should also be saved as a separate .csv file.

*  Save each file with a short, meaningful file name (see DRYAD recommendations here), except the README file which should just be called README.txt

*  save all image, audio, and video files in formats recommended by DRYAD (here). You may wish to contact DRYAD or your Editor if the raw data files are too large.

*  upload all of your files to DRYAD or other repository and fill in all of the metadata and information requested by the repository, even if this is not required as it makes your data easier to find and understand

*  from the repository get a URL that can be used by editors and reviewers before your data are made public with a DOI

 

 

last updated 7 September 2020

September 1, 2020

 The Editorial Board of The American Naturalist has completed an evaluation of a paper by Pruitt JN , Howell KA , Gladney SJ , Yang Y , Lichtenstein JLL , Spicer MElise , Echeverri SA, and Pinter-Wollman N . 2017. Behavioral Hypervolumes of Predator Groups and Predator-Predator Interactions Shape Prey Survival Rates and Selection on Prey Behavior . The American Naturalist. 189:254–266

https://www.journals.uchicago.edu/doi/abs/10.1086/663680

A committee of six individuals, including both Associate Editors and outside research community members, were appointed by the Editor, Dan Bolnick, in late February 2020. They delivered their report on this in April 2020. After several rounds of comment by Dr. Pruitt and his co-authors, the committee requested that the Dryad repository be updated. This update has been completed (https://datadryad.org/stash/dataset/doi:10.5061/dryad.8q8p7)  and the committee and this Editor have no further concerns about this paper.

The concern identified by the committee is summarized here:

Inconsistency between sea star sample sizes reported in the paper, and the data provided. The paper reports a sample size of 28 sea stars. But there are only 19 lines of personality data for the sea stars, although in some data files a seastar with ID #20 is listed. In figures reporting sea star behavioral repeatability data, there are either 19 or 20 visible data points (Figs. A2-3, and Fig. A4, respectively), and the statistics reported in the paper seem consistent with a sample size of 19.

Based on communications from Dr. Pinter-Wollman and Dr. Pruitt, it appears that:

1) not all collected individuals were used, hence the difference between 28 collected individuals and the actual sample size in the study. This point is slightly confusing for readers but not significant enough to require publication of a correction, as the Methods do not state that 28 individuals were used.
2) A 20th individual of unknown original provenance, obtained from a laboratory setting, was included in the study but not in the Dryad repository. The committee suggested, and the authors agreed, that this can be fixed by updating the Dryad data repository to include the 20th individual. Note that this entails the addition of a new version of the data file, not a replacement [the original version remains]. From our point of view, the correction statement is sufficient to clarify what was absent in the original data file, and why. The Dryad repository has now been corrected as of August 18 2020. The .csv file in the update has a header that reads:
"Not all 28 sampled sea stars were used for the study. Note the 20th individual used in this study came from a teaching collection."
The Editorial Board remains concerned that the use of one individual of unknown provenance, in a sample of otherwise wild-caught sea stars, is a less than optimal experimental design, as it may inflate trait variance in a study of sea star behavioral variation. The origin of the 20th individual was not made clear in the published manuscript methods, nor was the rationale for its addition explained. However, the concern was deemed relatively minor, by the committee, as we do not have grounds to believe this changed the core results of the paper. We therefore accept the correction to the Dryad repository as sufficient to satisfy the minor concerns raised regarding this paper. Unless other concerns come to light that we are not presently aware of, we consider this paper to be valid and suitable for citing in future work.

Signed,
Daniel Bolnick
Editor-In-Chief, The American Naturalist
September 1, 2020

August 31, 2020

PubPeer and ongoing evaluation of past published papers

In my capacity as Editor-In-Chief of The American Naturalist, I sometimes receive emails criticizing already-published papers in this journal.  The criticisms sometimes come in the form of anonymous emails to me. Other times they come from known emails and contain lengthy attachments with detailed criticisms. Increasingly, however, these criticisms are posted publicly on PubPeer, often by anonymous individuals. I want to take a moment to comment on this trend, from my perspective. Note that the following is not formal journal policy, and other Editors  (current and future) may disagree.

The individuals posting anonymously on PubPeer are typically genuinely concerned members of the scientific community who have identified errors, or strongly suspect errors exist, in the scientific record. Their criticisms should be addressed for their technical content. Left unanswered, a cloud hangs over the paper(s) in question.

That said, there is some question about the proper procedure for answering these criticisms. Yes, PubPeer itself leaves room for comments (interestingly, journal editors like myself must pay money to reply to comments, even if to acknowledge them and state we are evaluating the issue). But, this process bypasses the journal that publishes the paper, and bypasses the normal scientific tradition of external review by experts in the field chosen by the journal editor for their knowledge and hopefully objectivity. For this reason, I want to really encourage people with substantial concerns about a paper (e.g., which may appreciably alter the results and conclusions), to submit formal "Comments" (different journals call these different things) to the journal. These Comment manuscripts get reviewed (including by the original authors), and the original authors may write a Reply. Based on the reviews and comment and reply the Editor decides whether to (1) publish just the comment, (2) publish the comment and reply, (3) request a correction or retraction, or (4) let matters lie because the criticism does not have solid ground to stand on, is deemed to be debatable, or (per COPE guidelines) if the issue has no material impact on the conclusions of the paper. 

I know that the Comment approach is onerous in several ways. First, it takes time to prepare a formal manuscript formatted submission. I know of at least one instance (concerning a putative error in an equation in a theory paper) where the critic took time to write a pages-long missive, but said s/he didn't want to take the time to actually turn it into a Comment.  Second, the Comment is authored and so the critic identifies him or herself to possible reprisal. Recent activity by lawyers of one author (letters, FOIA requests) illustrates the point that critics can be subjected to legal and other harassment, which is a stressful penalty for free-speech in pursuit of ensuring scientific accuracy. {Note; anonymously authored Comments are not traditional, but might possibly be considered}. Third, Comments go through review which takes time, leaving the matter behind the scenes in the meantime. Here is where things get tricky: if a criticism proves to be groundless, a PubPeer criticism can damage an author's reputation needlessly and inappropriately. On the other hand, if a criticism has merit, we'd like it to be public knowledge immediately to correct the scientific record as soon as a problem is identified. These are conflicting needs. PubPeer is on the side of rapid dissemination, and reviewing Comments behind the scenes can err on the side of perpetuating errors too long (and can also open journals to criticisms of foot-dragging and cover-ups). While perhaps such cover-ups exist, every case I am directly aware of I have been satisfied that journals are genuine in their desire for ensuring quality science, but must exercise caution in leaping to conclusions without due process (which occurs behind the scenes).

So, as a journal, our official stance is that we prefer to receive submitted Comments. This follows the scientific tradition of obtaining third-party review, and deliberate evaluation of criticisms. But, scientific traditions are fluid and we are in an era of increasing speed and openness:  Preprint servers, open peer review, open data, et cetera. We therefore also recognize that PubPeer is an active tool in science conversations. The criticisms posted there can be valid identification of genuine problems that need to be evaluated formally and corrected. If valid well-justified and substantial concerns exist and are published on PubPeer, then the affected journal should respond.

There are caveats and concerns about PubPeer. Most importantly, it is important that these not be used as a mechanism for pursuing personal vendettas. Excessive targeting of an author with multiple minor complaints can constitute a kind of harassment, and may be viewed as such by University Equity officers or equivalent. The anonymous nature of many PubPeer comments makes it easier for impacted authors to feel like (and, argue that) they are the target of personal vendettas and harassment. Second, the existence of PubPeer comments can cast a long shadow over a paper whether the comments are profound or minor.  This shadow can affect an author's career prospects (fellowship applications, job applications, etc) even before the matter is resolved and judged to be valid or not. The result can be inappropriate damage to an innocent authors' career, which in turn may have grave consequences for mental health. Third, there is an established mechanism for voicing complaints about papers in science:  contact the author to request clarification, or contact the Editor, or submit a Comment. I realize that many complainants are nervous about revealing their identity, and these three steps I list make this harder. One can create new email accounts to anonymously contact an author, or Editor. But bear in mind that although scientific publication is in no way bound to follow legal court-style procedures, there's something to be said for the legal tradition that accused have a chance to face their accuser. Science publication is not a court of law; but it is worth considering whether we as a community want to respect that kind of tradition. That said, whistleblowers sometimes hide their identity for good reason (e.g., not trusting that whistleblower laws offer sufficient protection). With this in mind, it might be worth considering whether journals might publish anonymizied Comments. That runs counter to the growing trend towards open peer review, but can protect vulnerable individuals when their complaint is valid and they have compelling grounds to be concerned about how their career is impacted.

How should journals respond to PubPeer comments? This is something of a wild-west issue that I expect COPE (the committee on publication ethics) is grappling with to provide guidelines. First, the PubPeer comments need to be brought to the Editor's attention. Don't assume that Editors take time to search PubPeer regularly; at least I don't. Then, the journal begins an evaluation process that may include (1) forming a committee to evaluate the papers and the citicisms, and/or (2) contacting the author(s) for a response. These take time, indeed can take many months to do properly with due care for the details and reaching a scientifically defensible and just decision. So, please be patient. Sometimes there are delays due to valid extensions in recognition of health concerns, parental leave, etc, that need to be appropriately accommodated even in pursuit of ensuring scientific truth. Also, do not expect any formal journal response on the PubPeer website; they charge Editors money to make any statement as Editor (that's how they pay their bills, at least in part).

Obviously if there is compelling evidence of malpractice by authors, the investigation should ultimately result in retraction (e.g., a Comment alone ceases to be sufficient). But, some cases where a critic may reasonably suspect data have been fudged, may appear to others to be less clear-cut. A journal's job is to carefully evaluate the opposing claims and the weight of evidence. The result may end up being inaction where the journal's editor or their appointed committees are unconvinced by the criticisms. It is my belief that, if a criticism proves to be invalid, then its public airing on PubPeer can do more harm than good. Which is why I prefer such processes be handled via a formal evaluation and review.

To summarize, I believe that PubPeer has appealing features, but has the potential to do harm to innocent authors, or to be used for personal attacks. Our goal should be to promote its strengths while mitigating the risks.  As editor I therefore prefer that critiques begin with direct communication with an author, and that journals be notified via submission of a Comment when those direct communications do not yield a satisfactory answer (or, go unanswered). In this way, concerns can be evaluated and questions can be answered, through a formal review process that might resolve the issue without exposing the author to public ridicule that might ultimately prove to be baseless.  That said, I understand the motive for using PubPeer (anonymity, speed, openness) and will respond when made aware of PubPeer critiques that appear to have scientific merit.

I am aware of PubPeer criticisms targeting multiple papers of some authors. These cases take extra time to evaluate because they entail more work: committee members need to find time to read many papers, delve into their data, and authors need to respond to more comments. Please be patient. To the broader community, a journal may appear to be ignoring criticisms; but this may simply be the result of a careful and incomplete process. In some cases we also are awaiting COPE recommendations for how to proceed in cases that reveal new kinds of publication-ethics problems.

In particular, we are aware of cases where published papers lack complete data repositories (e.g., on Dryad). A present, we are awaiting guidance from the Committee on Publication Ethics on how to proceed in such cases.  This is a new enough situation, that the recommendations are not yet clearly articulated as a scientific community. As an interim policy, our first step is to notify authors and give them a deadline for completing their files to our satisfaction. Such deadlines may take some time, for instance if hard drive forensics are needed, or if an individual is on parental or health leave. If the data files cannot be completed by the deadline, our present plan is to issue Editorial Expressions of Concern. Authors with such EoCs will have any future papers scrutinized far more closely for complete data. Authors would do well to avoid such situations by uploading complete Dryad (or equivalent) archives following best practices recommended by Dryad for thorough meta-data, ReadMe files, etc.

 As an institution The American Naturalist is dedicated to publishing rigorous, well-supported research, and that rigor takes time. Every other Editor-In-Chief of other journals whom I have interacted with extensively shares that commitment. We will never sweep valid criticisms under the rug to defend our reputation, because that reputation comes not from having papers, but from the quality of the papers that we disseminate. But, sometimes the deliberate process of internal review may appear to the outside to be inaction; often nothing could be further from the truth.

Ultimately, it is journals that publish science (for now). And thus it is journals' job to correct errors in the science that they previously published. It follows that for any serious scientific error, it rests in journals' hands to evaluate the error and take suitable corrective steps.  Of course, if journals truly fail to meet that obligation, that's a problem. And PubPeer represents an alternative, a safety net for criticisms, and a way of shaming journals into action. But - I'll say this again - sometimes what looks like inaction from outside is either (1) a slow deliberative process to ensure complete and fair evaluation, or (2) a completed evaluation that found nothing to move forward with. So, please give journals some time to do what they need to do. 

Dan Bolnick


Addendum:

A Twitter comment pointed out that PubPeer enables author responses to critiques, creating online dialogues that may resolve and clarify and issue. I appreciate and value that. The poster (Boris Barbour) also notes that most PubPeer comments are not graced with author defenses or acknowledgement. If correct (no statistics were offered in support), that is troubling. And in my view that again points to the value of journals as the enforcers whose job is to follow through and either require a response or else publish a retraction or Expression of Concern or Comment. Authors can ignore PubPeer comments because... because they can, there's no immediate penalty for engaging besides the court of public opinion.