Well-calibrated confidence measures for multi-label text classification with a large number of labels. (February 2022)

• Record Type: Journal Article
• Authors: Maltoudoglou, Lysimachos; Paisios, Andreas; Lenc, Ladislav; Martínek, Jiří; Král, Pavel; Papadopoulos, Harris
• Is Part Of: Pattern recognition. Volume 122(2022)
• Publication Date: 2022-02
• Subjects: Text classification -- Multi-label -- Word2vec -- Bert -- Conformal prediction -- Label powerset -- Computational efficiency -- Nonconformity measure -- Confidence measure; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108271
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a novel approach to address the computationally demanding nature of the Label Powerset (LP) Inductive Conformal Prediction (ICP) multi-label classification with a high number of labels. We mathematically establish the validity of the proposed approach and provide experimental results that highlight its computational efficiency. We present prediction set results for data-sets in for multi-label text classification problems where it was previously computationally challenging and show that can be practically useful. Results show that Bert classifier surpasses the non-contextualised based by a large margin. In addition, to the best of our knowledge, our bert implementation achieved state-of-the-art results in the data-sets used. Abstract: We extend our previous work on Inductive Conformal Prediction (ICP) for multi-label text classification and present a novel approach for addressing the computational inefficiency of the Label Powerset (LP) ICP, arrising when dealing with a high number of unique labels. We present experimental results using the original and the proposed efficient LP-ICP on two English and one Czech language data-sets. Specifically, we apply the LP-ICP on three deep Artificial Neural Network (ANN) classifiers of two types: one based on contextualised (bert) and two on non-contextualised (word2vec) word-embeddings. In the LP-ICP setting we assign nonconformity scores to label-sets from which the corresponding p -values and prediction-sets are determined. Our approach deals with the increased computational burden of LP by eliminating from consideration a significant number of label-sets that will surely have p -values below the specified significance level. This reduces dramatically the computational complexity of the approach while fully respecting the standard CP guarantees. Our experimental results show that the contextualised-based classifier surpasses the non-contextualised-based ones and obtains state-of-the-art performance for all data-sets examined. The good performance of the underlying classifiers is carried on to their ICP counterparts without any significant accuracy loss, but with the added benefits of ICP, i.e. the confidence information encapsulated in the prediction sets. We experimentally demonstrate that the resulting prediction sets can be tight enough to be practically useful even though the set of all possible label-sets contains more than 1 e + 16 combinations. Additionally, the empirical error rates of the obtained prediction-sets confirm that our outputs are well-calibrated.