Confidence and precision increase with high statistical power

We are delighted that our Analysis article (Power failure: why small sample size undermines the reliability of neuroscience. Nature Rev. Neurosci. 14, 365–376 (2013))¹ has stimulated debate about the issues arising from low statistical power in neuroscience studies. Here, we take the opportunity to respond to some important points made by Quinlan (Misuse of power: in defence of small-scale science. Nature Rev. Neurosci. http://dx.doi.org/10.1038/nrn3475-c1 (2013))², Ashton (Experimental power comes from powerful theories — the real problem in null hypothesis testing. Nature Rev. Neurosci. http://dx.doi.org/10.1038/nrn3475-c2 (2013))³ and Bacchetti (Small sample size is not the real problem. Nature Rev. Neurosci. http://dx.doi.org/10.1038/nrn3475-c3 (2013))⁴, and clarify our position on certain key issues raised in our original article¹.

Quinlan² draws a comparison between our observations and those made by Friston in his recent article⁵. However, we note that subsequent commentaries by Ingre⁶ and by Lindquist and colleagues⁷ echo many of the concerns that we raised and that Friston⁸ agreed that he was “convinced by most of their observations”. The main concern — originally raised by Friston⁵ and reiterated by Quinlan² — is that high-powered studies will generate formally statistically significant differences for a 'trivial' effect. There are three important rejoinders to this concern. First, in studies with low statistical power, an observed effect will necessarily be large if it is to pass a pre-specified p-value threshold (typically 0.05), but this does not mean that the true effect will be large, or even exist at all. Second, the concern only applies if a p-value threshold (that is, typically 0.05) is used to either reject or (implicitly) accept the null hypothesis. As Ingre notes⁶, larger studies actually protect against inferences from trivial effect sizes by allowing a better estimation of the magnitude of the true effect. A shift in emphasis away from significance testing towards the use of effect sizes and confidence intervals would therefore improve matters — this is a point we return to below. Third, the true effect size is not known in advance; what is considered 'trivial' can only be determined when the effect size is known. For example, candidate gene studies are frequently quite small N studies, with the implicit assumption that common genetic variants will be associated with reasonably large effects. Over a decade of relatively fruitless research showed that this assumption is not correct; the magnitude of these effects turns out to be much smaller⁹, as has been borne out by more recent genome-wide association studies (GWASs), which are explicitly designed to be able to detect very small effects. Have these smaller effects turned out to be trivial? Not at all — in many cases, the genes that have been identified using well-powered GWAS methods have led to renewed interest in the role of the pathways implicated and have thereby generated genuinely new insights into disease mechanisms¹⁰. Moreover, learning that the effect sizes of common genetic variants on complex traits are very small has provided important fundamental insights into the genetic architecture of these traits¹¹. Therefore, high power provides greater precision in the estimation of the actual effect size so that researchers can assess their importance or triviality with confidence.