A comparison of methods for health policy evaluation with controlled pre‐post designs. (12th February 2020)

• Record Type: Journal Article
• Authors: O'Neill, Stephen; Kreif, Noemi; Sutton, Matt; Grieve, Richard
• Is Part Of: Health services research. Volume 55:Number 2(2020)
• Page Start: 328
• Page End: 338
• Publication Date: 2020-02-12
• Subjects: difference‐in‐differences -- interactive fixed effects -- pay‐for‐performance -- policy evaluation -- synthetic control; Medical care -- Periodicals; Medical care -- Evaluation -- Periodicals; Hospital care -- Periodicals; Health services administration -- Periodicals; 362
• DOI: 10.1111/1475-6773.13274
• Languages: English
• ISSNs: 0017-9124

Abstract

Abstract: Objective: To compare interactive fixed effects (IFE) and generalized synthetic control (GSC) methods to methods prevalent in health policy evaluation and re‐evaluate the impact of the hip fracture best practice tariffs introduced for hospitals in England in 2010. Data Sources: Simulations and Hospital Episode Statistics. Study Design: Best practice tariffs aimed to incentivize providers to deliver care in line with guidelines. Under the scheme, 62 providers received an additional payment for each hip fracture admission, while 49 providers did not. We estimate the impact using difference‐in‐differences (DiD), synthetic control (SC), IFE, and GSC methods. We contrast the estimation methods' performance in a Monte Carlo simulation study. Principal Findings: Unlike DiD, SC, and IFE methods, the GSC method provided reliable estimates across a range of simulation scenarios and was preferred for this case study. The introduction of best practice tariffs led to a 5.9 (confidence interval: 2.0 to 9.9) percentage point increase in the proportion of patients having surgery within 48 hours and a statistically insignificant 0.6 (confidence interval: −1.4 to 0.4) percentage point reduction in 30‐day mortality. Conclusions: The GSC approach is an attractive method for health policy evaluation. We cannot be confident that best practice tariffs were effective.