Pull back the curtain: External data validation is an essential element of quality improvement benchmark reporting. Issue 1 (July 2020)

• Record Type: Journal Article
• Authors: Jakubus, Jill Lynn; Di Pasquo, Shauna L.; Mikhail, Judy N.; Cain-Nielsen, Anne H.; Jenkins, Peter C.; Hemmila, Mark R.
• Is Part Of: Journal of trauma and acute care surgery. Volume 89:Issue 1(2020)
• Publication Date: 2020-07
• Subjects: Trauma outcomes -- quality improvement -- data validation -- interrater reliability -- trauma registry; Surgical intensive care -- Periodicals; Surgical emergencies -- Periodicals; Wounds and injuries -- Surgery -- Periodicals; 617.026
• DOI: 10.1097/TA.0000000000002579
• Languages: English
• ISSNs: 2163-0755

Abstract

Abstract : BACKGROUND: Accurate and reliable data are pivotal to credible risk-adjusted modeling and hospital benchmarking. Evidence assessing the reliability and accuracy of data elements considered as variables in risk-adjustment modeling and measurement of outcomes is lacking. This deficiency holds the potential to compromise benchmarking integrity. We detail the findings of a longitudinal program to evaluate the impact of external data validation on data validity and reliability for variables utilized in benchmarking of trauma centers. METHODS: A collaborative quality initiative-based study was conducted of 29 trauma centers from March 2010 through December 2018. Case selection criteria were applied to identify high-yield cases that were likely to challenge data abstractors. There were 127, 238 total variables validated (i.e., reabstracted, compared, and reported to trauma centers). Study endpoints included data accuracy (agreement between registry data and contemporaneous documentation) and reliability (consistency of accuracy within and between hospitals). Data accuracy was assessed by mean error rate and type (under capture, inaccurate capture, or over capture). Cohen's kappa estimates were calculated to evaluate reliability. RESULTS: There were 185, 120 patients that met the collaborative inclusion criteria. There were 1, 243 submissions reabstracted. The initial validation visit demonstrated the highest mean error rate at 6.2% ± 4.7%, and subsequent validation visits demonstrated a statistically significant decrease in error rate compared with the first visit ( p < 0.05). The mean hospital error rate within the collaborative steadily improved over time (2010, 8.0%; 2018, 3.2%) compared with the first year ( p < 0.05). Reliability of substantial or higher (kappa ≥0.61) was demonstrated in 90% of the 20 comorbid conditions considered in the benchmark risk-adjustment modeling, 39% of these variables exhibited a statistically significant ( p < 0.05) interval decrease in error rate from the initial visit. CONCLUSION: Implementation of an external data validation program is correlated with increased data accuracy and reliability. Improved data reliability both within and between trauma centers improved risk-adjustment model validity and quality improvement program feedback. Abstract : Supplemental digital content is available in the text.