Calibration of score based likelihood ratio estimation in automated forensic facial image comparison. (May 2022)

• Record Type: Journal Article
• Authors: Rodriguez, Andrea Macarulla; Geradts, Zeno; Worring, Marcel
• Is Part Of: Forensic science international. Volume 334(2022)
• Publication Date: 2022-05
• Subjects: Facial image comparison -- Calibration -- Deep learning -- Forensic science -- Score based likelihood ratio; Medical jurisprudence -- Periodicals; Chemistry, Forensic -- Periodicals; Forensic Medicine -- Periodicals; Médecine légale -- Périodiques; Chimie légale -- Périodiques; Gerechtelijke geneeskunde; Gerechtelijke chemie; Gerechtelijke psychiatrie; Chemistry, Forensic; Medical jurisprudence; Electronic journals; Periodicals; Electronic journals; 614.1
• DOI: 10.1016/j.forsciint.2022.111239
• Languages: English
• ISSNs: 0379-0738

Abstract

Abstract: Forensic facial image comparison lacks a methodological standardization and empirical validation. We aim to address this problem by assessing the potential of machine learning to support the human expert in the courtroom. To yield valid evidence in court, decision making systems for facial image comparison should not only be accurate, they should also provide a calibrated confidence measure. This confidence is best conveyed using a score-based likelihood ratio. In this study we compare the performance of different calibrations for such scores. The score, either a distance or a similarity, is converted to a likelihood ratio using three types of calibration following similar techniques as applied in forensic fields such as speaker comparison and DNA matching, but which have not yet been tested in facial image comparison. The calibration types tested are: naive, quality score based on typicality, and feature-based. As transparency is essential in forensics, we focus on state-of-the-art open software and study their power compared to a state-of-the-art commercial system. With the European Network of Forensic Science Institutes (ENFSI) Proficiency tests as benchmark, calibration results on three public databases namely Labeled Faces in the Wild, SC Face and ForenFace show that both quality score and feature based calibration outperform naive calibration. Overall, the commercial system outperforms open software when evaluating these Likelihood Ratios. In general, we conclude that calibration implemented before likelihood ratio estimation is recommended. Furthermore, in terms of performance the commercial system is preferred over open software. As open software is more transparent, more research on open software is urged for. Highlights: Automated Face Comparison systems can help the forensic investigator in court. Likelihood Ratio (LR) is an appropriate method to convey Face Comparison assessments. Applying filters to calibration improves the LR estimation compared to naive calibration. Commercial software performance outperforms open software.