Update on Modified Double Blind

After 10 months of our experiment in modified double blind, it seemed like a good time to provide a bit of an update on how it's going.

To recap, the journal committed to redacting the peer review system and removing the title page from PDFs but has left it up to the authors to what extent the authors want to anonymize their papers beyond that. If authors have chosen ways to reveal themselves, then we decided that, at this stage, double blind is essentially a service to authors and we wouldn't police that. Authors can also choose to opt out entirely, identifying themselves on the title page.

We've had to make a lot of new policies and judgment calls as unexpected twists developed. I think most of those were caused because the community isn't used to the concept--and there many ways in which authors have already identified themselves before submitting--e.g., preprints, Figshare or GitHub links, submissions to single-blind journals first. In addition, double-blind seemed to make reviewers oddly chatty about the act of reviewing, at least at first. It's gotten easier as people have gotten used to it. And fairly soon, we're hoping it will get easier since several more journals are talking about trying our version of modified double blind or even 100% compulsory double blind (neither reviewers or authors can choose to reveal themselves).

As the process settled down, we were able to provide a clearer set of instructions on the webpage about the ways authors might not be anonymous. Taking the last three months since that change, the percentage of papers whose authors have opted out is 16%. Most of them have done so because they felt their identities were too obvious to pretend otherwise. A few are simply against the idea that there might be a problem that double blind would fix.

One of the questions we had was whether reviewers would be more likely to turn down review requests if the identity of the authors is unknown, but the percentage who agree to review has stayed essentially the same. Only a few reviewers have refused explicitly because of double blind review. Some felt not knowing would alter the quality of the review. Some disagreed with the premise that there might be implicit bias.

The repeated objection to double-blind review in science is that author identities are almost always guessable, making it a pointless exercise. Not all guesses reported to us, however, have been accurate. Sometimes the lab was guessed but the lead author was not, so minimizing unconscious bias still seems to have value for that lead author. As one person summed up in a discussion of double blind on Twitter, “There’s a difference between guessing and knowing—I think that gap even if small is important.”

It's a bit too early to say whether the experiment has made a difference in what papers get accepted, but at least one editor thinks there's one noticeable difference: well-known people may be getting much tougher reviews.

Trish Morse
Managing Editor

Links about signing reviews

As our experiment in trying double blind lite continues, I found this post on Small Pond Science about signing or not signing reviews, with the extended comments, very interesting: http://smallpondscience.com/2014/03/26/why-i-prefer-anonymous-peer-reviews/

But I do have to say, as someone who reads all of the reviews that leave the office, anonymous reviewers have been no less diligent and professional than the reviewers who signed. And the very few that have serious problems are caught in the editorial process and addressed.

Three links mentioned in the comments are ones I've read and liked in the past, so I thought I'd repeat them here:

• Ten tips on how to write a good peer review in any field from the Guardian: http://www.theguardian.com/higher-education-network/blog/2013/sep/27/peer-review-10-tips-research-paper
• And though I have referred to this before, there is this extensive look at peer review and how to write a review from the British Ecological Society: http://www.britishecologicalsociety.org/wp-content/uploads/Publ_Peer-Review-Booklet.pdf
• Results of a survey that Molecular Ecology did about signing reviews: http://www.molecularecologist.com/2014/04/do-we-sign-our-peer-reviews-mostly-n 

Am Nat Switches to Double Blind Review in the New Year

Over the next few months, the American Naturalist will be making the transition to a form of double blind reviewing--reviewers will no longer be informed of the authors' identities in an effort to minimize explicit or implicit biases triggered by author names or institutions. As a speaker on the problem said, "Implicit bias produces 'micro-inequities'" that are not always easy to detect.

This transition began at Evolution 2014 in Raleigh, North Carolina, where workshops run by About Women in Science explored issues facing early career women. A presentation for the post-graduate workshop is here:
http://www.clfs.umd.edu/biology/dudashlab/Evol2014%20Post-Tenure%20Workshop%20ImplicitBias%20Final.pdf

And another on implicit bias here:
http://www.clfs.umd.edu/biology/dudashlab/Resources%20to%20learn%20more%202July2014.pdf

The editorial board of the American Naturalist met during the annual meeting and discussed ways we might address the problem of implicit bias. At that time, it was proposed that we try a form of double-blind review. We would blind the authors' identities in the system but not attempt to anonymize the paper itself.

Many people have commented that double-blind review won't work in the sciences, at least in its most exacting form. I tended to agree because I started out in publishing at a true double-blind social science journal and that's how I understood its workings. To anonymize a paper, authors were required to write their papers in such a way that their connection to their previous work was disguised. This produced some very unclear and often turgid writing since it required writing in a passive voice and in the third person to avoid indicating who had done the author's previous work in the same area. Reviewers would criticize the writing. Then after acceptance, the papers had to be thoroughly rewritten. This rewriting and re-rewriting is a labor-intensive process that puts the burden on the authors and requires extensive investments in journal office staff  (to police every page of 900 submissions and/or rewrite every successful manuscript after acceptance). However, the field itself sought anonymity and took the process of double blinding very seriously, in spite of the extra work. I'm using the past tense because even the humanities and social sciences are questioning whether double blind is "a fiction" in the "age of Google." https://www.insidehighered.com/news/2011/05/31/american_economic_association_abandons_double_blind_journal_reviewing

In science, there has always been the problem of how to disguise the origins of a paper from a particular lab or research group. The study organism, the methods, and the references to prior publications can make it all too easy to guess the principal investigator no matter how much a paper is rewritten. Research that has already been presented at conferences is readily searchable and might even have been heard by a reviewer. If authors post preprints, then a quick Google will reveal all. It's not enough to say that reviewers should be too ethical to Google the topic. Checking on the state of the topic or looking up particular references cited are natural parts of the reviewing process. Though not all papers would get "unmasked" in these ways, it does make rewriting and re-rewriting papers to achieve double blind purity a bad investment of everyone's time--and it would require considerable disruption of regular professional activity. If, however, the goal is to minimize unconscious bias, simply removing author names, email addresses, and institutions from title pages and correspondence may make a difference in leveling the playing field at very little cost. There is a claim that just increasing the uncertainty of authorship has a positive effect in leveling the playing field.
http://www.cs.utexas.edu/users/mckinley/notes/snodgrass-tods-2007.pdf

Increasing the use of forms of double-blind review in the sciences seems to be an idea that's spreading:

http://www.nature.com/news/journals-weigh-up-double-blind-peer-review-1.15564

And so, the American Naturalist will experiment with redacting author identities throughout the system and in reviewer correspondence to minimize bias. Starting in January 2015, authors will be asked to provide title pages with no author names, affiliations, or email addresses. Authors will instead provide that information inside the submission system. They will also be asked to upload their acknowledgments separately during review.

It's taken awhile to figure out how to convert the system. It's just going to have to be a bit messy for a few months! It will be an interesting New Year.


Trish Morse
Managing Editor