A new concordance measure for risk prediction models in external validation settings. (1st June 2016)

• Record Type: Journal Article
• Authors: van Klaveren, David; Gönen, Mithat; Steyerberg, Ewout W.; Vergouwe, Yvonne
• Is Part Of: Statistics in medicine. Volume 35:Number 23(2016)
• Page Start: 4136
• Page End: 4152
• Publication Date: 2016-06-01
• Subjects: concordance -- discrimination -- logistic regression -- proportional hazards regression -- case‐mix heterogeneity -- censoring; Medical statistics -- Periodicals; Statistique médicale -- Périodiques; Statistiques médicales -- Périodiques; 610.727
• DOI: 10.1002/sim.6997
• Languages: English
• ISSNs: 0277-6715

Abstract

Abstract : Concordance measures are frequently used for assessing the discriminative ability of risk prediction models. The interpretation of estimated concordance at external validation is difficult if the case‐mix differs from the model development setting. We aimed to develop a concordance measure that provides insight into the influence of case‐mix heterogeneity and is robust to censoring of time‐to‐event data. We first derived a model‐based concordance ( mbc ) measure that allows for quantification of the influence of case‐mix heterogeneity on discriminative ability of proportional hazards and logistic regression models. This mbc can also be calculated including a regression slope that calibrates the predictions at external validation ( c‐mbc ), hence assessing the influence of overall regression coefficient validity on discriminative ability. We derived variance formulas for both mbc and c‐mbc . We compared the mbc and the c‐mbc with commonly used concordance measures in a simulation study and in two external validation settings. The mbc was asymptotically equivalent to a previously proposed resampling‐based case‐mix corrected c‐index. The c‐mbc remained stable at the true value with increasing proportions of censoring, while Harrell's c‐index and to a lesser extent Uno's concordance measure increased unfavorably. Variance estimates of mbc and c‐mbc were well in agreement with the simulated empirical variances. We conclude that the mbc is an attractive closed‐form measure that allows for a straightforward quantification of the expected change in a model's discriminative ability due to case‐mix heterogeneity. The c‐mbc also reflects regression coefficient validity and is a censoring‐robust alternative for the c‐index when the proportional hazards assumption holds. Copyright © 2016 John Wiley & Sons, Ltd.