Abstract
In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
209,00 € per year
only 17,42 € per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Gelman, A. & Stern, H. The difference between “significant” and “not significant” is not itself statistically significant. Am. Stat. 60, 328–331 (2006).
Rosnow, R.J. & Rosenthal, R. Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44, 1276–1284 (1989).
Loftus, G.R. & Masson, M.E.J. Using confidence intervals in within-subject designs. Psychon. Bull. Rev. 1, 476–490 (1994).
Cumming, G., Fidler, F. & Vaux, D.L. Error bars in experimental biology. J. Cell Biol. 177, 7–11 (2007).
Henson, R. What can functional neuroimaging tell the experimental psychologist? Q. J. Exp. Psychol. A 58, 193–233 (2005).
Poldrack, R.A. et al. Guidelines for reporting an fMRI study. Neuroimage 40, 409–414 (2008).
Rosenthal, R. & Gaito, J. The interpretation of levels of significance by psychological researchers. J. Psychol. 55, 33–38 (1963).
Author information
Authors and Affiliations
Contributions
S.N. conceived the project and made the figure. S.N., B.U.F. and E.-J.W. conducted the literature analyses and wrote the paper.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Rights and permissions
About this article
Cite this article
Nieuwenhuis, S., Forstmann, B. & Wagenmakers, EJ. Erroneous analyses of interactions in neuroscience: a problem of significance. Nat Neurosci 14, 1105–1107 (2011). https://doi.org/10.1038/nn.2886
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.2886
This article is cited by
-
Sexual Orientation Affects Neural Responses to Subtle Social Aggression Signals
Archives of Sexual Behavior (2024)
-
Effect of hypnosis versus mindfulness on self-defined treatment goals and sleep following acquired brain injury: a randomized actively controlled trial
Discover Psychology (2024)
-
Functional connectivity abnormalities of brain networks in obsessive–compulsive disorder: a systematic review
Current Psychology (2024)
-
Individual attractiveness preferences differentially modulate immediate and voluntary attention
Scientific Reports (2023)
-
Cross-hemispheric recruitment during action planning with increasing task demand
Scientific Reports (2023)
