A Within-Study Approach to Evaluating the Role of Moderators of Impact in Limiting Generalizations from "Large to Small". (March 2022)

• Record Type: Journal Article
• Authors: Jaciw, Andrew P.; Unlu, Fatih; Nguyen, Thanh
• Is Part Of: American journal of evaluation. Volume 43:Number 1(2022)
• Page Start: 108
• Page End: 131
• Publication Date: 2022-03
• Subjects: within-study comparison -- generalizability -- RCT -- effect replication -- moderator effects; Evaluation -- Periodicals; Public administration -- Evaluation -- Periodicals; Educational evaluation -- Periodicals; Social sciences -- Methodology -- Periodicals; 300.72
• DOI: 10.1177/10982140211030552
• Languages: English
• ISSNs: 1098-2140

Abstract

There is a burgeoning body of evidence on the average impacts of educational programs. Yet, for many local decision makers, because impacts can vary across sites, the question of whether a certain program will work in their particular district or school remains. This article addresses the question of the generalizability of large-scale average causal effects to specific locales, that is, from "large to small." Our method evaluates whether impacts measured on the large-scale generalize to individual sites and whether adjusting for variation among sites in their characteristics accounts for impact heterogeneity that limits generalization. Our application to a multisite experimental evaluation in education shows that the average impact does not readily generalize to sites, with discrepancies between 0.17 and 0.41 standard deviations, and that characteristics of sites account only minimally for this variation. This emphasizes need for caution by local decision makers in accepting average impact findings as generalizable to their specific locales.