RD or Not RD: Using Experimental Studies to Assess the Performance of the Regression Discontinuity Approach. (February 2018)

• Record Type: Journal Article
• Authors: Gleason, Philip; Resch, Alexandra; Berk, Jillian
• Other Names: Wong Vivian C. guest-editor.; Steiner Peter M. guest-editor.
• Is Part Of: Evaluation review. Volume 42:Number 1(2018)
• Page Start: 3
• Page End: 33
• Publication Date: 2018-02
• Subjects: methodological development -- content area -- quasi-experimental design -- methodology (if appropriate) -- outcome evaluation (other than economic evaluation) -- design and evaluation of programs and policies -- education; Evaluation research (Social action programs) -- Periodicals; 361.0072
• DOI: 10.1177/0193841X18787267
• Languages: English
• ISSNs: 0193-841X

Abstract

Background: This article explores the performance of regression discontinuity (RD) designs for measuring program impacts using a synthetic within-study comparison design. We generate synthetic RD data sets from experimental data sets from two recent evaluations of educational interventions—the Educational Technology Study and the Teach for America Study—and compare the RD impact estimates to the experimental estimates of the same intervention. Objectives: This article examines the performance of the RD estimator with the design is well implemented and also examines the extent of bias introduced by manipulation of the assignment variable in an RD design. Research design: We simulate RD analysis files by selectively dropping observations from the original experimental data files. We then compare impact estimates based on this RD design with those from the original experimental study. Finally, we simulate a situation in which some students manipulate the value of the assignment variable to receive treatment and compare RD estimates with and without manipulation. Results and conclusion: RD and experimental estimators produce impact estimates that are not significantly different from one another and have a similar magnitude, on average. Manipulation of the assignment variable can substantially influence RD impact estimates, particularly if manipulation is related to the outcome and occurs close to the assignment variable's cutoff value.