A Bayesian evaluation framework for subjectively annotated visual recognition tasks. (March 2022)

• Record Type: Journal Article
• Authors: Prijatelj, Derek S.; McCurrie, Mel; Anthony, Samuel E.; Scheirer, Walter J.
• Is Part Of: Pattern recognition. Volume 123(2022)
• Publication Date: 2022-03
• Subjects: Uncertainty estimation -- Epistemic uncertainty -- Supervised learning -- Bayesian inference -- Bayesian modeling; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108395
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Addresses problems in current evaluation procedures for subjectively annotated problems in computer vision. Proposes a framework for evaluation of any black box predictor's epistemic uncertainty through Bayesian modeling of P Y ^ ( Y ^ | Y ) . Applies the framework to 4 practical use cases of subjectively annotated tasks where humans provide subjective annotations. Graphical abstract: Abstract: An interesting development in automatic visual recognition has been the emergence of tasks where it is not possible to assign objective labels to images, yet still feasible to collect annotations that reflect human judgements about them. Machine learning-based predictors for these tasks rely on supervised training that models the behavior of the annotators, i.e., what would the average person's judgement be for an image? A key open question for this type of work, especially for applications where inconsistency with human behavior can lead to ethical lapses, is how to evaluate the epistemic uncertainty of trained predictors, i.e., the uncertainty that comes from the predictor's model. We propose a Bayesian framework for evaluating black box predictors in this regime, agnostic to the predictor's internal structure. The framework specifies how to estimate the epistemic uncertainty that comes from the predictor with respect to human labels by approximating a conditional distribution and producing a credible interval for the predictions and their measures of performance. The framework is successfully applied to four image classification tasks that use subjective human judgements: facial beauty assessment, social attribute assignment, apparent age estimation, and ambiguous scene labeling.