Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty. (November 2017)

• Record Type: Journal Article
• Authors: Anderson, Samantha F.; Kelley, Ken; Maxwell, Scott E.
• Is Part Of: Psychological science. Volume 28:Number 11(2017)
• Page Start: 1547
• Page End: 1562
• Publication Date: 2017-11
• Subjects: statistical power -- sample size -- effect size -- publication bias -- methodology; Psychology -- Periodicals; 150.5
• DOI: 10.1177/0956797617723724
• Languages: English
• ISSNs: 0956-7976

Abstract

The sample size necessary to obtain a desired level of statistical power depends in part on the population value of the effect size, which is, by definition, unknown. A common approach to sample-size planning uses the sample effect size from a prior study as an estimate of the population value of the effect to be detected in the future study. Although this strategy is intuitively appealing, effect-size estimates, taken at face value, are typically not accurate estimates of the population effect size because of publication bias and uncertainty. We show that the use of this approach often results in underpowered studies, sometimes to an alarming degree. We present an alternative approach that adjusts sample effect sizes for bias and uncertainty, and we demonstrate its effectiveness for several experimental designs. Furthermore, we discuss an open-source R package, BUCSS, and user-friendly Web applications that we have made available to researchers so that they can easily implement our suggested methods.