First off, it's from Nature Communications, a much lower tier part of the NPG family (impact factor 12.4 vs. 40+ for the real deal). Secondly, it really depends what you're trying to show to determine whether this is problematic. Generally for animal studies you do a pre hoc analysis to determine sample size. But the dirty secret is that everyone's power analysis is garbage - you generally don't know the expected variability in your data, you almost certainly don't know the expected differences you'll be observing, and choosing a power of 0.8 and alpha of 0.05 is more or less arbitrary (albeit standard).Originally Posted by Loki
"we continuously increased the number of animals until statistical significance was reached to support our conclusions."
https://www.nature.com/articles/s41467-017-02765-w.pdf
Exact quote from an article published in Nature, arguably the top journal in the natural sciences. WTF?
Doing the kind of study reported here is valuable if you're really just trying to get information about the population(s) you're studying - essentially, the size of an effect and the variability in your data. Studies published in the likes of Nature Communications often provide this level of evidence. It's not great for hypothesis testing, but that doesn't mean it isn't okay science. Hell, pretty much all of particle physics is just smashing stuff together enough times that you get a signal that meets a given set of criteria.
The big issue here was whether they were running multiple statistical tests on the data without providing adjustments for p-values afterwards. This is a very common sin, and accounting for it is crucial to make sure you're not just reporting noise instead of real data. It might indeed be a big problem but without a clearer understanding of what they did it's hard to know.
Reply With Quote