Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions. (7th June 2019)

• Record Type: Journal Article
• Authors: Orr, Larry L.; Olsen, Robert B.; Bell, Stephen H.; Schmid, Ian; Shivji, Azim; Stuart, Elizabeth A.
• Is Part Of: Journal of policy analysis and management. Volume 38:issue 4(2019)
• Page Start: 978
• Page End: 1003
• Publication Date: 2019-06-07
• Subjects: Policy sciences -- Periodicals; Political planning -- United States -- Periodicals; United States -- Social policy -- Periodicals; 353
• DOI: 10.1002/pam.22154
• Languages: English
• ISSNs: 0276-8739

Abstract

Abstract: Evidence‐based policy at the local level requires predicting the impact of an intervention to inform whether it should be adopted. Increasingly, local policymakers have access to published research evaluating the effectiveness of policy interventions from national research clearinghouses that review and disseminate evidence from program evaluations. Through these evaluations, local policymakers have a wealth of evidence describing what works, but not necessarily where. Multisite evaluations may produce unbiased estimates of the average impact of an intervention in the study sample and still produce inaccurate predictions of the impact for localities outside the sample for two reasons: (1) the impact of the intervention may vary across localities, and (2) the evaluation estimate is subject to sampling error. Unfortunately, there is relatively little evidence on how much the impacts of policy interventions vary from one locality to another and almost no evidence on the implications of this variation for the accuracy with which the local impact of adopting an intervention can be predicted using findings from an evaluation in other localities. In this paper, we present a set of methods for quantifying the accuracy of the local predictions that can be obtained using the results of multisite randomized trials and for assessing the likelihood that prediction errors will lead to errors in local policy decisions. We demonstrate these methods using three evaluations of educational interventions, providing the first empirical evidence of the ability to use multisite evaluations to predict impacts in individual localities—i.e., the ability of "evidence‐based policy" to improve local policy.