Peer Review

[wiggin]

As many of you have likely heard, there was a kerfluffle in the stem cell research world in the last few months. A very high profile paper was published in a very high impact journal (Nature) purporting to show that somatic cells could be reprogrammed to stem cells using very simple stimuli, including treatment by a light acid bath. These 'STAP' cells (or stimulus triggered acquisition of pluripotency) were a very big deal - if verified in humans, it would mean that very simple methods could be used to turn any cells into pluripotent stem cells, opening up all sorts of therapeutic avenues. (Currently, acquired pluripotency to make IPS cells is a very complex process that is not likely to be easily replicated in a clinical situation.)

Immediately after publication, much of the stem cell world was skeptical. I remember that after I heard about the paper, while chatting with a friend in January, I immediately responded with disbelief and assumed there were artefactual issues with the data. I was hardly the only one; immediately all of the top stem cell labs in the world went about trying to replicate the data. No one met with success, and within weeks a growing controversy (along with some irregularities in the manuscript including image duplication and plagiarism from the author's previous work) let to some co-authors suggesting the paper should be retracted, and a full investigation was launched by the journal. The paper was given an unusual amount of benefit of the doubt, however, because the senior author on the paper was Charles Vacanti, a highly respected researcher at Harvard and Brigham and Women's Hospital who has a consistent track record of high impact, important publications.

As the months went on, however, it became clear that no one else was able to replicate the data. In June, the lead Japanese author on the project (who had started as a postdoc in Vacanti's lab, but published from RIKEN, a research instituted in Japan) agreed to retract the paper. Just a few weeks ago, another Japanese co-author (and her current supervisor) committed suicide, and RIKEN was undergoing a detailed investigation of the whole thing. Meanwhile, Vacanti's long and top-notch time as chairperson of anesthesiology at BWH was cut short, and his career is effectively ruined (though I doubt he knew anything was fishy).

So far, this seems like a tragic story but an effective validation of the scientific community. High profile results result in skepticism, which deepens upon the failure of multiple labs to replicate the results. In time, the research is discredited or proven, and science moves forward. And yet, there was a critical flaw in this whole chain of events - peer review and editorial review of the paper did not catch the flaws in the study.

But wait, you say! Peer review can't determine whether a study is replicable, just whether the data as presented appear to be complete and thorough, and the authors drew appropriate conclusions from the data. And the relatively minor issues with figure replication etc. are very challenging to detect on a simple peer review without overwhelming scrutiny of every publication ever published by the authors. So how is this a failure? I think it's a failure because of two anecdotes surrounding this: First, the aforementioned fact that I, a lowly postdoc who doesn't even work directly with IPS cells, instantly didn't believe the study. Second, my boss is the editor in chief of another high profile journal, and scuttlebutt says his journal FIRST rejected the paper on the grounds that they didn't believe the data was representative of reality, either. So if it didn't pass the sniff test for a bunch of people (the paper was rejected by a bunch of journals before finally being accepted at Nature, after much revision), why did the peer reviewers let it pass?

I think the answer is that journals like Nature are hungry to publish the newest, most highly cited papers, and that if this HAD panned out as real, it would have been yet another remarkable coup for one of the three best scientific journals in the world. Editors and reviewers all view 'unbelievable' data as a good thing because it will get a lot of readership and could quite possibly be paradigm shifting research. All too often, though, in the rush to publication journals get sloppy and allow flashy research to get published even if it maybe should be subjected to more strict review.

I think I also run into this issue when I'm carrying out peer review of publications. I always don't want to come off as a dick (even if peer review is blinded), but I try to give detailed, cogent critiques of the paper - from the motivation to choice of methods to data that should have been reported but wasn't. But sometimes I just don't believe a bit of data. It's not that I suspect dishonesty, but I think researchers can introduce artifacts or otherwise misinterpret their data, and convince themselves they've found something amazing. How should a peer reviewer approach this problem? Should they assume the data as presented are replicable and leave the policing of replicability to after publication? Or should they follow up with very detailed questions about methodologies and requests for additional studies just based on a vague suspicion that the data looks too good to be true? This is an issue I've wrestled with, and I'm curious what you guys think. What is the real role and responsibility of the peer reviewer in the scientific process?

[Loki]

I'll address this in more detail later, but surely it's the reviewers and not Nature per se that allowed this to go through? Depending on the reviews, the editor might have felt that their hands were tied. This could have very well gotten published because of sheer luck in the manuscript getting gullible (or simply incompetent) reviewers.

[wiggin]

Nah, editors always get the final say - they didn't even have to send it out for peer review, let alone accept a manuscript they had reservations about. And I can't imagine that they're idiots and didn't realize how controversial this study was likely to be. They would (or should) not have sent it out to a 'gullible' reviewer of Nature. And the chances of getting 3+ gullible Nature reviewers are beyond miniscule.

Lots of other good journals passed on this publication. I'm not singling Nature out for criticism - Science is equally guilty of this, as are other high impact publications - but I think that peer reviewers are not the only ones at fault here. In fact, since peer reviewers tend to be asked to review under a certain set of assumptions (i.e. that the data presented is real and reproducible), I'd say these kind of judgments are more in the purview of the editor than that of the individual reviewer. That doesn't absolve a reviewer of raising questions if the data seems too good to be true, but I think a lot of responsibility rests with the editors of these journals to provide some healthy skepticism about data quality.

[Aimless]

When you and your boss look back at your sniff-tests, what aspects of the two papers stand out as being particularly dodgy--eg. likely artefacts--and why?

[Loki]

Nah, editors always get the final say - they didn't even have to send it out for peer review, let alone accept a manuscript they had reservations about. And I can't imagine that they're idiots and didn't realize how controversial this study was likely to be. They would (or should) not have sent it out to a 'gullible' reviewer of Nature. And the chances of getting 3+ gullible Nature reviewers are beyond miniscule.

Lots of other good journals passed on this publication. I'm not singling Nature out for criticism - Science is equally guilty of this, as are other high impact publications - but I think that peer reviewers are not the only ones at fault here. In fact, since peer reviewers tend to be asked to review under a certain set of assumptions (i.e. that the data presented is real and reproducible), I'd say these kind of judgments are more in the purview of the editor than that of the individual reviewer. That doesn't absolve a reviewer of raising questions if the data seems too good to be true, but I think a lot of responsibility rests with the editors of these journals to provide some healthy skepticism about data quality.

I don't know about your field, but the standard for not sending something out to peer review in poli sci is that the manuscript either doesn't deal with the issues that the journal cares about or is just of a vastly inferior quality. And while an editor can theoretically reject any manuscript even if all reviewers say it's great, editors usually defer to the reviewers (unless the reviewers disagree with one another).

[wiggin]

I don't know about your field, but the standard for not sending something out to peer review in poli sci is that the manuscript either doesn't deal with the issues that the journal cares about or is just of a vastly inferior quality. And while an editor can theoretically reject any manuscript even if all reviewers say it's great, editors usually defer to the reviewers (unless the reviewers disagree with one another).

I think editors have a lot more leeway, at least in science/engineering. Yes, they generally screen for scope/relevance and quality (along with impact) before sending out for review, but they can also reject it if they think the results are too good to be true. Obviously that can lead to 'groupthink', where accepted dogma can't get overturned, but editors only use this prerogative in rare cases. Once sent to peer review, though, it's likely that the same questions will be raised, and that truly revolutionary results will get scrutinized for flaws in methodology. It's extremely common for reviewers - and, in turn, editors - to ask for extensive additional experiments to verify the results prior to publication in a high impact journal. In some cases, they've even required independent verification of the results (in one famous case, Nature published a paper in the late 80s purporting to show 'water memory' was true - a basic theory underlying homeopathy - but in a remarkable departure from usual procedures, included a disclaimer that the editors and reviewers didn't really believe the data and were doing an independent review of the methodology and data). Editors in high profile journals have a LOT of leeway to reject papers, either before peer review or after positive reviews, mostly because there's a long line of other deserving research that could get published in its stead. I'm not talking about podunk journals that will get a bad reputation if they turn down deserving papers - these are journals with impact factors in the 30s and 40s that everyone reads. They have a lot of leverage and aren't obliged to publish - the reason they DO publish, however, is that they don't want to get scooped on the hottest paper of the year. Peer reviewers don't have the same conflict of interest, of course, but also I'm not sure if it's really their job to assess the believability or reproducibility of the results if the methodology and data look sound.

[Unheard Of]

There's a lot wrong with the way science is published (at least from what I see in biology), the STAP paper is just a particularly extreme example. This recent discussion between some leading genomics experts brings up a few more. I wish I knew a better system to fight for

[Dreadnaught]

On a related note, this makes me wonder if there's a limitation to the whole idea of academic publication. IE, just putting these things on paper can only determine "whether the data as presented appear to be complete and thorough, and the authors drew appropriate conclusions from the data." But, in this multimedia world, maybe we should be insisting on a higher standard of evidence because said evidence is easier than ever to record.

Or am I just bringing things down to the lowest common denominator?

[Unheard Of]

On a related note, this makes me wonder if there's a limitation to the whole idea of academic publication. IE, just putting these things on paper can only determine "whether the data as presented appear to be complete and thorough, and the authors drew appropriate conclusions from the data." But, in this multimedia world, maybe we should be insisting on a higher standard of evidence because said evidence is easier than ever to record.

Or am I just bringing things down to the lowest common denominator?

For our projects, we press our partners to put all data used (be it for a paper or in the projects 'deliverables') into public archives (SRA,GEO,ArrayExpress,whatever). Sometimes we have to use controlled access archives (dbGaP being an example), but we only do this if the way human donors consented requires it. If there isn't an archive we put the data out on an FTP site. Where possible we place biological materials into some form of biobank, if they aren't there already. I aim to have every analysis method documented to the nth degree and intend to explore docker-ising our analysis pipelines. The objective is to make all the data and materials available, and all the methods openly accessible, so that the work can be reproduced and extended. We're largely funded by public money, so it's not like any of this stuff really belongs to us. If someone needs to mail me to get the data or get a clear picture of our methods, I've fucked up.

Sadly, some of our partners are using analysis software that they have developed themselves and either haven't published or haven't released publicly (or both). Others feel that their data is commercially valuable. I find this deeply depressing, and hope they all find their careers hindered by their attitudes.

Does this meet your standards Dread'?

There's a lot wrong with the way science is published (at least from what I see in biology), the STAP paper is just a particularly extreme example. This recent discussion between some leading genomics experts brings up a few more. I wish I knew a better system to fight for

On the other hand, our paper was accepted by Science this week, so everything to do with academic publishing is clearly fine

[Loki]

http://www.nature.com/news/social-sc...n-bias-1.15787 Speaking of peer review: it seems the problem starts at a much earlier stage (where scientists are deciding whether to submit a manuscript for review or even whether to write one up).

Also: http://www.nature.com/news/replicati...d-copy-1.10634

[wiggin]

On a related note, this makes me wonder if there's a limitation to the whole idea of academic publication. IE, just putting these things on paper can only determine "whether the data as presented appear to be complete and thorough, and the authors drew appropriate conclusions from the data." But, in this multimedia world, maybe we should be insisting on a higher standard of evidence because said evidence is easier than ever to record.

Or am I just bringing things down to the lowest common denominator?

Certainly more and more people are making their raw data available publicly (at least after publication), and journals have instituted stricter rules for e.g. image manipulation of gels/microscopy. Yet it's still very hard to evaluate the data during peer review without spending more time than any reviewer actually has... and said analysis will still only be as good as the data. Methodological problems (which are often quite subtle and only come up during attempts at replication) won't be captured.

For our projects, we press our partners to put all data used (be it for a paper or in the projects 'deliverables') into public archives (SRA,GEO,ArrayExpress,whatever). Sometimes we have to use controlled access archives (dbGaP being an example), but we only do this if the way human donors consented requires it. If there isn't an archive we put the data out on an FTP site. Where possible we place biological materials into some form of biobank, if they aren't there already. I aim to have every analysis method documented to the nth degree and intend to explore docker-ising our analysis pipelines. The objective is to make all the data and materials available, and all the methods openly accessible, so that the work can be reproduced and extended. We're largely funded by public money, so it's not like any of this stuff really belongs to us. If someone needs to mail me to get the data or get a clear picture of our methods, I've fucked up.

Sadly, some of our partners are using analysis software that they have developed themselves and either haven't published or haven't released publicly (or both). Others feel that their data is commercially valuable. I find this deeply depressing, and hope they all find their careers hindered by their attitudes.

Does this meet your standards Dread'?

Impressive! I think this works for some areas of research but not others. Example - large scale -omics studies, or particle physics stuff like the data on arXiv. Yet lots of research is essentially bespoke, lab-scale data and there really isn't that much more to put online. Oh, you can show all of the extra side data you've accumulated, replicated experiments, more extensive sets of histological/etc. data - and you should - but publicly archived data and methods won't necessarily solve that much. My own lab does a lot of work developing new materials, which involve lots of dead ends, optimization of e.g. reaction conditions, etc. - most of the data is largely useless to anyone, poorly searchable, and just adds more to the pile (and it's not publication bias to exclude it, generally, either). Sometimes curation has its points.

On the other hand, our paper was accepted by Science this week, so everything to do with academic publishing is clearly fine

Congratulations!!!! That's absolutely fantastic news. Very big deal. I'm afraid that the highest I aspire to is the likes of Nature Materials.

http://www.nature.com/news/social-sc...n-bias-1.15787 Speaking of peer review: it seems the problem starts at a much earlier stage (where scientists are deciding whether to submit a manuscript for review or even whether to write one up).

Also: http://www.nature.com/news/replicati...d-copy-1.10634

I think the problems with methodology and publication bias in psychological studies has been well documented. I'm very skeptical of any data coming out of that field lately given the apparent systematic problems in how research is conducted. I tend to be much more confident in biological data, though, because most of the methods used are well established and shouldn't have significant methodological problems. The best papers, though, use new techniques and technologies that may have subtle (or not-so-subtle) drawbacks, and even well-validated older techniques can be mishandled by inexperienced researchers.

[Dreadnaught]

For our projects, we press our partners to put all data used (be it for a paper or in the projects 'deliverables') into public archives (SRA,GEO,ArrayExpress,whatever). Sometimes we have to use controlled access archives (dbGaP being an example), but we only do this if the way human donors consented requires it. If there isn't an archive we put the data out on an FTP site. Where possible we place biological materials into some form of biobank, if they aren't there already. I aim to have every analysis method documented to the nth degree and intend to explore docker-ising our analysis pipelines. The objective is to make all the data and materials available, and all the methods openly accessible, so that the work can be reproduced and extended. We're largely funded by public money, so it's not like any of this stuff really belongs to us. If someone needs to mail me to get the data or get a clear picture of our methods, I've fucked up.

Sadly, some of our partners are using analysis software that they have developed themselves and either haven't published or haven't released publicly (or both). Others feel that their data is commercially valuable. I find this deeply depressing, and hope they all find their careers hindered by their attitudes.

Does this meet your standards Dread'?

Yes'm.

[Loki]

http://retractionwatch.com/2015/05/2...ta-were-faked/

Same story in non-sciency source: http://www.politico.com/story/2015/0...131.html?ml=po

And some real world implications: http://kieranhealy.org/blog/archives...-consequences/