Scientists Aim To Pull Peer Review Out Of The 17th Century

The technology that drives science forward is forever accelerating, but the same can’t be said for science communication. The basic process still holds many vestiges from its early days — that is the 17th century.

Some scientists are pressing to change that critical part of the scientific enterprise.

Here’s what they’re confronting: When researchers studying the biology of disease make a discovery, it typically takes nine months for them to get their results published in a journal.

One reason for that delay is it goes through a process of peer review that is both necessary and antiquated. The fate of that paper rests on just two or three scientists who have been asked to review it and decide whether it’s worthy of being published.

Imagine how this would feel if the matter in question were a consumer product.

“If the only thing Amazon ever published were reviews of the first three people who bought a product, then we’d have a very ineffective system for knowing what was good and bad,” says Michael Eisen, a Howard Hughes Medical Institute (HHMI) investigator at University of California, Berkeley.

The scientific reviews may be carried out by someone with a personal ax to grind, or people who aren’t trained to evaluate a study’s methods or statistics. And to top it off, reviews are usually anonymous and kept secret by the journal that commissioned it.

“If you buy pen refills on Amazon, you get far more useful feedback about the benefits and deficits of a particular product than you do about a work of science that represents years and years of peoples’ work and millions of dollars of public investment,” Eisen says.

He is not alone in his concerns. Eisen recently attended a meeting of biomedical researchers who want to find a way to modernize this process, to make it more fitting for a world that now lives online and isn’t so concerned about the price of paper stock for printing presses.

Part of the problem the community confronts is that peer review isn’t simply about providing feedback to scientists so they can improve the papers they submitted.

“Often peer review is being asked to do two things,” says Theo Bloom, executive editor of the BMJ (formerly the British Medical Journal). “It’s being asked to look at technical quality and to look at whether an article is suitable for a particular journal.”

Bloom says peer review does a reasonable job of picking studies of interest to journals such as hers, but it does a poor job of improving the quality of the paper.

Some years ago, for instance, she says scientists sent around papers with nine deliberate errors in them. Peer reviewers generally found just three.

One major question the attendees considered was whether to make peer reviews available to all readers. The BMJ, unlike most journals that publish biomedical research, already does that. The identities of the reviewers are even made public — a suggestion that caused some consternation at the meeting.

But that doesn’t solve problems of research quality.

Dr. Michael Lauer, a deputy director of the National Institutes of Health, told the meeting that one huge problem is that scientists often fail to ask a fundamental question before starting an experiment, which is how many people or animals should be included in a study. By the time a study is done and submitted for publication, it’s too late.

“Peer review is not going to help,” he told the scientists. “It doesn’t matter how transparent the peer review is, we’re not going to be able to solve this problem.”

Another concern is that today scientists are judged primarily by which journal publishes their work. The greatest rewards tend to go to scientists who can get their papers into major journals such as Science, Nature and Cell. It matters less what the actual findings are.

“To me that is one of the very biggest problems in the system today,” says Erin O’Shea, president of HHMI, “and it drives a lot of behavior — behavior that we don’t want.”

And if a top journal reviews a paper and decides not to publish it, the scientist needs to start the process of getting to print all over again — which is a waste of resources. “Science moves slower because research isn’t available immediately,” O’Shea said at the meeting. “To me, those are big problems.”

One way forward is to encourage scientists to make their work publicly available on the Internet before it has been peer-reviewed or accepted in a journal. Biologists are starting to do that, using a preprint server called bioRxiv. Physicists have been doing this for years.

Peer review can come later, Eisen says. In fact, he’s putting together a system that will facilitate that, and it’s set to debut this summer.

“What we want to see happen next is to allow the scientists who are reading papers [as part of their normal work] … to review them,” he says.

As he envisions it, “you post a work, people comment on it, you update it, and if it gets better through that process, that’s great — now you’ve produced something good,” he says. “If, through the process of review and assessment, you and the community realize the work wasn’t right, it just sorts of fades and you mark it as such. And I think we’ll all be better off if that happens.”

Notably, it will change the incentives for publication. Right now, because scientists are rewarded for getting papers past peer review in the top journals, attention-grabbing findings are valued over careful work.

Eisen says his system of review before publication would flip those priorities.

“The fact that people are going to be reviewing and assessing the work … should provide a very strong incentive back to authors to try to produce work that’s actually reproducible and durable, rather than work that is just flashy and gets into the right journal,” Eisen tells Shots.

Among the challenges he faces is that scientists will struggle to find the time to do careful reviews of papers published online. They may also be tempted to game the system. But people who sign their name to their reviews, or are certified by a trusted body, would have an incentive not to simply shower praise on studies posted by friends and colleagues.

“You don’t want to develop a reputation as someone who just says nice things about your buddy,” Eisen says.

Posting articles prior to peer review raises other issues, as well.

“There’s a worry that the Wild West version [where] everyone posts what they like will lead to a lot more garbage out there that people can’t make out what to make of it,” says BMJ editor Bloom.

So she still sees a role for journalists, if not as gatekeepers, at least as curators. “I think probably for a while we’ll see a parallel track where the journals still provide some imprint of thorough review, statistical review, methodological review,” she says, while preprints will “allow the community to share more with each other much more rapidly.”

Copyright 2018 NPR. To see more, visit http://www.npr.org/.