The Effect of Sample Heterogeneity and Risk Categorization on Area Under the Curve Predictive Validity Metrics. (January 2017)

• Record Type: Journal Article
• Authors: Howard, Philip D.
• Other Names: Helmus L. Maaike guest-editor.; Babchishin Kelly M. guest-editor.
• Is Part Of: Criminal justice and behavior. Volume 44:Number 1(2017:Jan.)
• Page Start: 103
• Page End: 120
• Publication Date: 2017-01
• Subjects: predictive validity -- violence risk assessment -- actuarial risk prediction -- AUC -- area under the curve -- female offenders -- OGRS -- OVP -- OASys; Correctional psychology -- Periodicals; 364.6
• DOI: 10.1177/0093854816678899
• Languages: English
• ISSNs: 0093-8548

Abstract

Forensic researchers often assume that the widely used area under the curve (AUC) predictive validity statistic can be readily compared across risk assessment instruments and between studies. From risk distributions for 224, 771 convicted English and Welsh offenders, I quantify the extent to which the AUCs of two continuously scored actuarial instruments are dependent upon sample heterogeneity and risk categorization. Sample composition can cause predictive validity to vary by 10% to 20% over chance between subpopulations, with higher AUCs for female subgroups whose scores have high variability, and lower AUCs when sampling is restricted by index offense and/or sentence type. Risk categorization has a potentially large effect on AUCs, which fall when few categories are used, especially when those categories contain unequal numbers of offenders. An improved understanding of how these factors can affect AUCs will inform professionals using risk assessment instruments and researchers comparing the validity of multiple instruments.