Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers. Issue 2 (3rd April 2018)

• Record Type: Journal Article
• Authors: Porter, Kristin E.
• Is Part Of: Journal of research on educational effectiveness. Volume 11:Issue 2(2018)
• Page Start: 267
• Page End: 295
• Publication Date: 2018-04-03
• Subjects: statistical power -- multiple hypothesis testing -- multiple testing procedures; Education -- Research -- Periodicals; Education -- Evaluation -- Periodicals; Education -- Methodology -- Periodicals; 371.422
• DOI: 10.1080/19345747.2017.1342887
• Languages: English
• ISSNs: 1934-5747

Abstract

ABSTRACT: Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs) are statistical procedures that counteract this problem by adjusting p values for effect estimates upward. Although MTPs are increasingly used in impact evaluations in education and other areas, an important consequence of their use is a change in statistical power that can be substantial. Unfortunately, researchers frequently ignore the power implications of MTPs when designing studies. Consequently, in some cases, sample sizes may be too small, and studies may be underpowered to detect effects as small as a desired size. In other cases, sample sizes may be larger than needed, or studies may be powered to detect smaller effects than anticipated. This paper presents methods for estimating statistical power for multiple definitions of statistical power and presents empirical findings on how power is affected by the use of MTPs.