Transparency and Open Science

This fall I had an extended conversation with Tim Parker from Whitman College, who is Co-organizer of the “Tools for Transparency in Ecology and Evolution” group. Here’s a brief summary of the TTEE goals:

“In November 2015, representatives (mostly editors-in-chief) from more than 20 journals in ecology and evolution joined researchers and funding agency panelists to identify ways to improve transparency in these disciplines. This workshop (funded by the US National Science Foundation and by the Laura and John Arnold Foundation and hosted by the Center for Open Science) identified general principles and specific tools that journals can adopt to encourage greater transparency of the science they publish. Most of the ideas that emerged from the workshop rest within the recently developed Transparency and Openness Promotion (TOP) framework (https://cos.io/top/). At this time, TOP contains eight separate editorial guidelines for journals, each designed to be adoptable by any empirical discipline”

Tim and I discussed the eight TOP guidelines, whose details can be found here . Here’s a summary of where we stand on this and where I would like to go in the coming year. I would welcome thoughts on this process from you.

1.     Citation Standards: AmNat is already at the highest “Level 3”, which stipulates that “ Article is not published until providing appropriate citation for data and materials” (e.g., a doi address for the data on Dryad or elsewhere), with rare exceptions granted for cases where authors get waivers regarding posting data (e.g., for ongoing analyses of long-term datasets, or for human data where privacy requirements prohibit data sharing).

2.     Data Transparency: AmNat is at the basic “Level 1” (“Article states whether data are available and, if so, where to access them.”). The higher Level 2 requires that all data are available without exception, which I do not think is feasible (e.g., human genetic studies often prohibit data sharing for privacy concerns as part of human subjects regulations). We are 95% of the way there, but that last 5% I don’t think is likely to happen.

3.     Code Transparency: We will implement the basic “Level 1” (the article states whether code used in analysis or simulation is publicly available, and if so where). If the community supports the move, I would be okay with shifting toward Level 2 in the near future (“Code must be posted to a trusted repository. Exceptions must be identified at article submission.”). I find it odd, by the way, that Code transparency allows exceptions while data transparency does not. My own unfortunate experience (which I wrote about here as a means of catharsis) is that code can contain mistakes. Transitioning to code repository will help encourage authors to proofread their code as carefully as they proofread their prose, thereby (hopefully) catching any minor code mistakes that can have major effects on conclusions.

4.     Materials Transparency: the reagents and materials that we use are so heterogeneous as a set of disciplines, that I did not see particular merit to requiring that the “Article states whether materials are available and, if so, where to access them.” So we will not presently aim for this standard.

5.     Design and Analysis Transparency: We will soon institute Level 1 (“Journal articulates design transparency standards.”). This will entail a checklist (e.g., confirming that the paper clearly reports sample sizes, effect sizes, which statistical tests, etc) that authors fill out upon resubmission of a revised manuscript. Higher levels require these checklists before we even review a paper the first time, which I think is a disincentive for submissions. Having this happen upon resubmission is a good way to help guide authors through careful revision. The checklist is currently a work in progress, building on a TTEE template.

6.     Study Plan Pre-registration: Pre-registration is the practice of publicly reporting one’s experimental design in advance of executing the project and publishing. This allows readers to be confident that the published work was indeed designed for the stated goal, rather than some post-hoc reinterpretation. I see the value in this goa, but I also strongly believe that biology often surprises us with un-looked for insights. Consequently, I have no interest in requiring pre-registration. Neither do I see a down-side to allowing authors to state whether and where they pre-registered. TOP’s “Level 1” requires that papers merely state whether or not the study design was pre-registered. If so, they state where the pre-registered plan can be accessed. There would be no penalty for studies that were not pre-registered.

7.     Analysis Plan Pre-registration: As with plan pre-registration, authors have the opportunity to record their statistical plans before obtaining data. This reduces the temptation to ‘P-hack’, doing analyses until a significant result is found. Again, I see value in allowing our data to surprise us and lead us to new questions, so I do not intend to require pre-registered analysis. But, I am fine with papers reporting whether or not they were pre-registered. So, I am planning on adopting TOP’s Level 1 standard: papers state whether or not the analyses were pre-registered, and if so, where the pre-registration can be found. Post-hoc analyses are still allowed, and manuscripts without pre-registration would still have equal footing with those that did pre-register.

8.     Replication: TOP’s replication guideline states that the basic level is that the “Journal encourages submission of replication studies.” This would be a big break from The American Naturalist’s tradition, as our brand is based on conceptual novelty and so generally we do not accept studies that duplicate a prior result in a new species, let alone in the same species as a prior study. Consequently, at present I do not intend to adopt this guideline.

I am entirely open to feedback on whether and how to institute these TOP guidelines. In particular, if anyone wants to help me craft some of the details (e.g., pre-registration statement formats, checklists), just drop me a note.

An additional topic often comes up when we discuss open science, which is the topic of author page charges, publication fees, versus open access. The American Naturalist is a non-profit journal published by the University of Chicago Press, unlike many of the larger journals published by for-profit institutions. We have converged on a hybrid model of publication. Authors who have sufficient resources to pay for Open Access fees (that help defray the costs of production, see this blog post for an explanation), are welcome to do so. But many of our papers are written by graduate students or junior researchers who may not have substantial funds for publication charges. We seek to accommodate these authors as well, by allowing people to publish non-open access and pay very low page charges. People who truly don't have the financial means to pay even these small charges can get waivers. By taking this model, we remove barriers-to-entry for aspiring authors who maybe can't afford several-thousands-of-dollars publishing costs seen at some other journals. We also make it possible for authors with means to go Open Access to reach a broader audience.

- Dan Bolnick

Incoming Editor-In-Chief

An open letter from the incoming AmNat Editor, Dan Bolnick

I wrote following message to send Associate Editors of The American Naturalist, to introduce myself as the incoming Editor-In-Chief and to discuss some matters of journal policy. In the interests of transparency, I am making a lightly edited version of the message public to everyone. I hope this helps clarify aspects of journal policy, and as a result proves useful to aspiring authors and our readers.  -Dan
_________________________________________________________________________________


Dear Associate Editors,

On January 1, Judie Bronstein will formally step down as Editor-In-Chief of The American Naturalist, and I will do my best to fill her role. I’d like to take this opportunity to thank her for her many years of service to our journal. Judie began as an Associate Editor in 2004, became an Editor in 2010, and Editor-In-Chief in 2013.  She’s set a high bar for thoughtful and constructive decisions. She oversaw several big initiatives, including the transition to double-blind review, and a series of 150^th anniversary articles highlighting the impact of past papers on our field today. Judie, thanks for your leadership, and enjoy a well-deserved break (once the papers you’ve been handling work their way through the system). I also want to extend a heartfelt ‘thanks’ to the staff at the University of Chicago Press who produce a wonderfully professional final product. You can read about who they are and what they do in a recent blog post (http://ecoevoevoeco.blogspot.com/2017/10/the-secret-lives-of-manuscripts.html ). And of course, thanks to all of you, the AEs who contribute valuable time and thought to identifying promising manuscripts, and working with the authors and reviewers to hone them into well-crafted papers.

            While I have met many of you in person over the years, or via email, I want to take a minute to introduce myself. I’ve been an AE of AmNat since 2008, and briefly also served as AE for Evolution. I like to think of myself as an “integrative biologist”, which is conveniently ambiguous. I tend to publish more in Evolution than Ecology, and go to the Evolution meetings more than ESA, though I occasionally go to the latter, so I suppose I’m an evolutionary ecologist with an emphasis on the former. In practice, my research at various times has touched on foraging behavior, mate choice, predator-prey dynamics, ecological causes of selection, speciation genetics, parallel evolution, biomechanics, genomics, parasitology, and immunology. I like to use a mix of research approaches, combining field observational natural history with field and lab experiments, genetics, meta-analysis, and theory. I suppose that’s why I’ve always gravitated towards The American Naturalist, for its interdisciplinary focus and its aspiration to fuse theory and data. On a more personal note, I’m a parent of two scientifically-minded young girls: the older is obsessed with the Big Bang and spectroscopy, the younger with lichens. My wife, Deborah Bolnick, is a human geneticist specializing in ancient DNA studies of North America. In the coming year we’ll be leaving Texas (where I’ve been at UT Austin for 13 years), for the cooler climate of Connecticut (University of…), to be closer to family and snow and mountains. I'm excited to join the strong EEB program at UConn, which also has a strong group of faculty working on microbiomes and on immunology. 

            While I have everyone’s attention, I also want to go over some important information that concerns all Associate Editors.  This is a bit long because there’s a lot I want to convey, so please bear with me:

Soliciting submissions: The number of journals serving ecology evolution and behavior has increased substantially in recent years. This trend has resulted in declining submission rates for many older society journals, including The American Naturalist. To combat this trend we need to be proactive about soliciting good submissions. I would like to ask each of you to proactively encourage colleagues to submit their work to The American Naturalist.  If you see a particularly good seminar, conference talk, or graduate student thesis defense, please consider suggesting our journal. I especially want to see us expand into fast-growing subjects where we have been a relatively minor player (e.g., genomics, among others).

            We don’t publish a large number of papers per issue (about 12 on average), so any given topic may be rarely covered. This gives our audience the false impression that “Am Nat doesn’t publish X anymore” (replace X with the researchers’ pet topic). To address this, Judie began a “Recent Articles in The American Naturalist” list, which is posted in the monthly ASN news letter and which gets a lot of hits online. I want to continue to fight these misconceptions by emphasizing that we remain conceptually broad, and welcome any high-quality paper in organismal biology broadly defined. You are a key part of spreading the word on this, by encouraging submissions in a wide range of subjects.  If you encounter a colleague who thinks their field is under-represented at AmNat, please point them to the Recent Articles lists (which have links to the cited articles). They’ll be surprised by the range of things we do publish. If they really don’t see their pet topic, then encourage them to try us anyway.

We especially encourage people to consider us for Syntheses and Perspectives; we get relatively few good proposals for these. The key is that an S&P needs to be more than a review, it needs to provide some novel insight(s) that emerge from a consideration of the existing literature.

            Many authors under-value their own work. In the past year I have colleagues say their  very-interesting results weren’t “AmNat-ty enough”. I fear we often miss outstanding papers because authors put our journal on a pedestal and don’t think their paper measures up. This self-deprecating bias may be especially severe for students. Please encourage authors who might otherwise self-select out of publishing with us.

           

Value added editorial decisions. A lot of journals send authors decision letters that basically rubber-stamp the reviews. I want to remind you that The American Naturalist aims to meet a higher bar. We want our AE and Editorial decision letters to add their own additional insights that add value to the paper. Whether it is published with us or not, we want to make the authors feel like we helped them out and made their eventual paper better. This helps us build a community of appreciative authors who are more likely to submit again, and more likely to read and cite our articles. This is more work, which is why we tend to send each AE fewer manuscripts than you might get as an AE at some other journals, but it is a key part of our brand.

Speed. I’ve seen multiple twitter threads suggesting that papers should take no more than 2 weeks from submission to decision.  I disagree. I think that papers should be returned as quickly as we can, but the priority is to render detailed and constructive commentary that improves the paper whatever the eventual decision. That takes time. I want to make it a hallmark of our journal that we prioritize careful constructive review, rather than sacrificing this on the altar of speed for its own sake. That said, the journal gets a very bad reputation (and loses submissions) when we take too long.  So, it is crucial that, when you are first assigned a paper, you look it over quickly (preferably within 1-2 days) and send it to review or write an editorial decline decision letter within one week (preferably less). Likewise, when reviews come back in, please write your decision letter within a week (preferably less). As AE, I typically sent a paper out to review (or not) within 48 hours, and tried to write a decision letter within 3 days of receiving reviews back.

 Editorial declines. If you judge that a paper just isn’t sufficiently engaging for our journal, please feel free to write an Editorial decline. But, there’s nothing more off-putting to an author than a half-baked editorial decline. If you opt for an Editorial decline, please write a serious review that gives the authors a road map for how they might improve the paper enough to be suitable for another journal (you may suggest other journals, also). We should aspire to add value to any manuscript that crosses our desk, and to be kind and thoughtful in our decisions. Our AEs are quite good at this, so we often get thank-you letters from authors after editorial declines, because our AEs and Editors put in a great deal of effort to be constructive even when declining, and because we render timely decisions this way. Editorial declines should take no more than a week. We have had some cases of editorial declines taking 3-4 weeks, which is unacceptably long and drives away authors. I know that detailed decline letters take time, and we are all busy.

Decline Without Prejudice. There is often some confusion about the boundaries between Decline, Decline without prejudice (DWOP) and Revise. Here’s a brief summary of my view on this.

We should decline when the paper has zero chance of being published with us. It may just not be novel or interesting enough, or may have fundamental flaws that we are confident cannot be fixed (e.g., an experimental design mistake that would require a complete re-do).

We should ask for revisions when the authors have failed to make a compelling case, but we suspect they can fix the paper through substantial rewriting, re-analysis, or by adding available data. When writing a ‘revise’ decision we want to make it very clear that we are not promising the paper will eventually be accepted. Rather, we should convey that if the authors can fix X, do Y, or add Z, to our satisfaction, then we will reconsider the paper.

I think most papers should really fall in either the ‘revise’ or ‘decline’ category. DWOP inhabits the grey area between these. I think DWOP should be used when we are very skeptical that the authors could successfully fix X, do Y, or add Z. Or, DWOP may be appropriate when we are demanding a fundamentally new element (new data, restructured model) to the paper whose outcome is very uncertain but crucial for acceptance. Conversely, we shouldn’t use DWOP as a substitute for “lots of revisions needed”. That can be made clear in how we write a ‘revision’ decision letter.

When to re-review? Journals are facing a lot of pressure to accelerate the time from submission to publication. One way to do this is to bypass a second round of reviews. When we receive a revision, it is the AE’s job to carefully examine the authors’ response to reviews, and read through the resubmitted manuscript. You have the right to decline the paper outright if they did not respond effectively to reviews, or to accept the paper if they did respond effectively. In fact, the default option should be for you to evaluate the resubmission. Sending a paper out for more review is needed only if  (1)  the authors have added new data or analyses that you are not qualified to evaluate, or (2) the authors have substantially rewritten a very large proportion of the paper (or fundamentally changed multiple figures), so that the text requires more than one pair of eyes.

Special Features. Until now, The American Naturalist has only published Special Issues arising from the American Society for Naturalists’ Vice Presidential Symposium that takes place at the Evolution meeting each year. We will be expanding to publish more Special Features (keep your eye out for one in the March 2018 issue, for instance). These will not be separately bound issues, but rather sections within a regular monthly issue. At first these will mostly arise from conference symposia sponsored by the American Society of Naturalists. In addition to the VP Symposium, ASN runs one extra symposium at the Evolution meeting, and at the Ecological Society of America meeting. We may also consider unsolicited proposals for special features. If you know of a potential special feature, talk to me about it to learn how we handle these. I want to emphasize a key point, however. In my experience reading Special Features in other journals, there are often papers that might not otherwise be good enough to see the light of day. I do not intend for us to compromise our standards for novelty or quality. Special Feature submissions may be rejected just like any submitted manuscript. If too few special feature articles pass through our usual filters at an acceptable pace, then we will publish the few that do as regular articles, reserving special features for batches of at least four papers on a coordinated topic. Unlike some journals, our special features will not be edited by the special feature organizers (who have a vested interest in the outcome).

Diversifying the AE pool. At present, 14 of our 64 AEs are female, and a large majority reside within the United States. This is, nevertheless, a big improvement over the distant past, because Judie and other recent co-Editors (Yannis Michilakis, Alice Winn, Troy Day, among others) have made a real effort to diversify our AE pool. One of my goals as Editor is to continue to improve the diversity of AEs. The good news is that women constitute 10 of the 19 new AEs who have started in 2017 or will start in 2018 (many appointed by Judie, before I came on the scene), and 8 of the 19 are non-US residents. I especially welcome suggestions for adding AEs from under-represented geographic regions (Latin America, Africa, Asia).

            Our AEs are typically drawn from people who have a proven track record of publishing and (more importantly) reviewing for us. We look for reviewers who give perceptive, in-depth, and timely comments. This means that the diversity of our AEs is partly constrained by the diversity of our reviewers (and authors). So, I want to encourage you to make an effort to recruit diverse reviewers (and solicit paper submissions from them). To help you with this, visit https://diversifyeeb.wordpress.com , and I especially encourage you to see the Blog post on this topic by your fellow AE Meghan Duffy: https://dynamicecology.wordpress.com/2017/09/06/choosing-reviewers-recognition-not-recall-and-why-lists-like-diversifyeeb-are-useful/

Easy Submission. We want to make it as simple as possible for authors to submit papers for review, which means stream-lining some of our requirements for review. The one thing I most want to bring to your attention is that we do not require authors format their manuscript to look like an AmNat paper (citation format, headings style, etc), at the initial submission. If the paper is worthwhile and its conceptual content satisfies us and the reviewers, there is time to bring it into nit-picky format later. So on initial review don’t worry about (for example), citations not being in exact AmNat style (though we request some form of (Author, year) format in the text.

Thanks for your attention, and I look forward to working with you all. I want you to know that I am very grateful for your help, and I am always open to questions and suggestions regarding the journal’s policies and directions.

Best wishes,

Dan Bolnick (incoming Editor-In-Chief)