An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials. (October 2016)

• Record Type: Journal Article
• Authors: Jaciw, Andrew P.; Lin, Li; Ma, Boya
• Other Names: Kelcey Ben guest-editor.; Spybrook Jessaca guest-editor.
• Is Part Of: Evaluation review. Volume 40:Number 5(2016)
• Page Start: 410
• Page End: 443
• Publication Date: 2016-10
• Subjects: education -- design and evaluation of programs and policies; Evaluation research (Social action programs) -- Periodicals; 361.0072
• DOI: 10.1177/0193841X16659600
• Languages: English
• ISSNs: 0193-841X

Abstract

Background: Prior research has investigated design parameters for assessing average program impacts on achievement outcomes with cluster randomized trials (CRTs). Less is known about parameters important for assessing differential impacts. Objectives: This article develops a statistical framework for designing CRTs to assess differences in impact among student subgroups and presents initial estimates of critical parameters. Research design: Effect sizes and minimum detectable effect sizes for average and differential impacts are calculated before and after conditioning on effects of covariates using results from several CRTs. Relative sensitivities to detect average and differential impacts are also examined. Subjects: Student outcomes from six CRTs are analyzed. Measures: Achievement in math, science, reading, and writing. Results: The ratio of between-cluster variation in the slope of the moderator divided by total variance—the "moderator gap variance ratio"—is important for designing studies to detect differences in impact between student subgroups. This quantity is the analogue of the intraclass correlation coefficient. Typical values were .02 for gender and .04 for socioeconomic status. For studies considered, in many cases estimates of differential impact were larger than of average impact, and after conditioning on effects of covariates, similar power was achieved for detecting average and differential impacts of the same size. Conclusions: Measuring differential impacts is important for addressing questions of equity, generalizability, and guiding interpretation of subgroup impact findings. Adequate power for doing this is in some cases reachable with CRTs designed to measure average impacts. Continuing collection of parameters for assessing differential impacts is the next step.