Improving the Use of Aggregate Longitudinal Data on School Performance to Assess Program Effectiveness: Evidence from Three Within Study Comparisons. Issue 3 (2nd July 2020)

• Record Type: Journal Article
• Authors: Hallberg, Kelly; Williams, Ryan; Swanlund, Andrew
• Is Part Of: Journal of research on educational effectiveness. Volume 13:Issue 3(2020)
• Page Start: 518
• Page End: 545
• Publication Date: 2020-07-02
• Subjects: Comparative interrupted time series -- quasi-experimental design -- within study comparison -- causal inference; Education -- Research -- Periodicals; Education -- Evaluation -- Periodicals; Education -- Methodology -- Periodicals; 371.422
• DOI: 10.1080/19345747.2019.1698088
• Languages: English
• ISSNs: 1934-5747

Abstract

Abstract: More aggregate data on school performance is available than ever before, opening up new possibilities for applied researchers interested in assessing the effectiveness of school-level interventions quickly and at a relatively low cost by implementing comparative interrupted times series (CITS) designs. We examine the extent to which effect estimates conducted using a CITS approach correspond to a randomized trial, varying (1) the approach to modeling the pre-intervention trend, (2) the approach used to select comparison cases, and (3) the number of years between the implementation of the intervention and measurement of the outcome of interest. Our analyses demonstrate that CITS can come close to replicating experimental findings. However, analytic decisions matter. Picking a comparison group that accounts for local context, past performance, and school demographics can reduce bias and reliance on model selection. Likewise, fitting nonlinear trend models can lead to substantial bias in short time series analyses. Finally, the performance of CITS declines for outcomes measured further after the introduction of the intervention.