Appropriateness of performance indices for imbalanced data classification: An analysis. (June 2020)

• Record Type: Journal Article
• Authors: Mullick, Sankha Subhra; Datta, Shounak; Dhekane, Sourish Gunesh; Das, Swagatam
• Is Part Of: Pattern recognition. Volume 102(2020:Jun.)
• Publication Date: 2020-06
• Subjects: Imbalanced classification -- Performance evaluation indices -- Precision -- Recall -- GMean -- Area under the curve; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107197
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Applicability of popular indices on imbalanced classification problem is analyzed. Two fundamental constraints are proposed to evaluate the appropriateness of an index. Remedial modifications are suggested in case of violation of constraint. A third proposed condition validates the quality of information returned by an index. Theoretical findings are empirically validated through extensive experiments. Abstract: Indices quantifying the performance of classifiers under class-imbalance, often suffer from distortions depending on the constitution of the test set or the class-specific classification accuracy, creating difficulties in assessing the merit of the classifier. We identify two fundamental conditions that a performance index must satisfy to be respectively resilient to altering number of testing instances from each class and the number of classes in the test set. In light of these conditions, under the effect of class imbalance, we theoretically analyze four indices commonly used for evaluating binary classifiers and five popular indices for multi-class classifiers. For indices violating any of the conditions, we also suggest remedial modification and normalization. We further investigate the capability of the indices to retain information about the classification performance over all the classes, even when the classifier exhibits extreme performance on some classes. Simulation studies are performed on high dimensional deep representations of subset of the ImageNet dataset using four state-of-the-art classifiers tailored for handling class imbalance. Finally, based on our theoretical findings and empirical evidence, we recommend the appropriate indices that should be used to evaluate the performance of classifiers in presence of class-imbalance.