Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation. Issue 3 (September 2018)

• Record Type: Journal Article
• Authors: Aczel, Balazs; Palfi, Bence; Szollosi, Aba; Kovacs, Marton; Szaszi, Barnabas; Szecsi, Peter; Zrubka, Mark; Gronau, Quentin F.; van den Bergh, Don; Wagenmakers, Eric-Jan
• Is Part Of: Advances in methods and practices in psychological science. Volume 1:Issue 3(2018)
• Page Start: 357
• Page End: 366
• Publication Date: 2018-09
• Subjects: nonsignificant results -- NHST -- Bayes factor analysis -- open data -- open materials -- preregistered; Psychology -- Periodicals; Psychology -- Research -- Periodicals; 150
• DOI: 10.1177/2515245918773742
• Languages: English
• ISSNs: 2515-2459

Abstract

In the traditional statistical framework, nonsignificant results leave researchers in a state of suspended disbelief. In this study, we examined, empirically, the treatment and evidential impact of nonsignificant results. Our specific goals were twofold: to explore how psychologists interpret and communicate nonsignificant results and to assess how much these results constitute evidence in favor of the null hypothesis. First, we examined all nonsignificant findings mentioned in the abstracts of the 2015 volumes of Psychonomic Bulletin & Review, Journal of Experimental Psychology: General, and Psychological Science ( N = 137). In 72% of these cases, nonsignificant results were misinterpreted, in that the authors inferred that the effect was absent. Second, a Bayes factor reanalysis revealed that fewer than 5% of the nonsignificant findings provided strong evidence (i.e., BF01 > 10) in favor of the null hypothesis over the alternative hypothesis. We recommend that researchers expand their statistical tool kit in order to correctly interpret nonsignificant results and to be able to evaluate the evidence for and against the null hypothesis.