On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study. Issue 3 (May 2021)

• Record Type: Journal Article
• Authors: Cunha, Washington; Mangaravite, Vítor; Gomes, Christian; Canuto, Sérgio; Resende, Elaine; Nascimento, Cecilia; Viegas, Felipe; França, Celso; Martins, Wellington Santos; Almeida, Jussara M.; Rosa, Thierson; Rocha, Leonardo; Gonçalves, Marcos André
• Is Part Of: Information processing & management. Volume 58:Issue 3(2021)
• Publication Date: 2021-05
• Subjects: Text classification -- Comparative study -- Systematic review; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2020.102481
• Languages: English
• ISSNs: 0306-4573

Abstract

Abstract: This article brings two major contributions. First, we present the results of a critical analysis of recent scientific articles about neural and non-neural approaches and representations for automatic text classification (ATC). This analysis is focused on assessing the scientific rigor of such studies. It reveals a profusion of potential issues related to the experimental procedures including: (i) use of inadequate experimental protocols, including no repetitions for the sake of assessing variability and generalization; (ii) lack of statistical treatment of the results; (iii) lack of details on hyperparameter tuning, especially of the baselines; (iv) use of inadequate measures of classification effectiveness (e.g., accuracy with skewed distributions). Second, we provide some organization and ground to the field by performing a comprehensive and scientifically sound comparison of recent neural and non-neural ATC solutions. Our study provides a more complete picture by looking beyond classification effectiveness, taking the trade-off between model costs (i.e., training time) into account. Our evaluation is guided by scientific rigor, which, as our literature review shows, is missing in a large body of work. Our experimental results, based on more than 1500 measurements, reveal that in the smaller datasets, the simplest and cheaper non-neural methods are among the best performers. In the larger datasets, neural Transformers perform better in terms of classification effectiveness. However, when compared to the best (properly tuned) non-neural solutions, the gains in effectiveness are not very expressive, especially considering the much longer training times (up to 23x slower). Our findings call for a self-reflection of best practices in the field, from the way experiments are conducted and analyzed to the choice of proper baselines for each situation and scenario. Highlights: A critical literature review reveals serious experimental issues in the recent ATC (neural) literature. We provide a very comprehensive and scientifically sound comparison of neural and non-neural methods. We consider a cost-effectiveness tradeoff analysis based on more than 1500 measurements. Simpler and cheaper non-neural solutions beat neural network methods in smaller datasets with a shortage of training. Transformer architectures are better in larger datasets but by small margins and at a much higher cost. Metafeatures are competitive with neural networks in both scenarios with a potentially better tradeoff.