Weight attention layer‐based document classification incorporating information gain. Issue 1 (27th September 2021)

• Record Type: Journal Article
• Authors: Lee, Min Seok; Yang, Seok Woo; Lee, Hong Joo
• Is Part Of: Expert systems. Volume 39:Issue 1(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-09-27
• Subjects: attention mechanism -- document classification -- feature selection -- information gain -- weight attention layer; Expert systems (Computer science); 006.33
• DOI: 10.1111/exsy.12833
• Languages: English
• ISSNs: 0266-4720

Abstract

Abstract: The performance of document classifiers largely depends on their internal representations of text data. Recent studies have been conducted to identify areas of focus and find latent data spaces to increase the representativeness and the performance of classifiers. In this study, we propose a weight attention layer (WAL) that uses an additional feature of words when computing their attention weights for deep learning models based on attention mechanisms. In the WAL, the attention distribution is calculated through the dot product of the attention weight matrix and a word weight matrix. We utilized information gain, which is one of the feature selection algorithms for the additional feature. To evaluate the proposed method, datasets of helpful reviews, sentiment reviews, and fake reviews were used. These datasets were applied to two deep learning models based on attention mechanisms, including an attention‐based bidirectional long short‐term memory (LSTM) and a hierarchical attention network. As a result of 10‐fold cross validation, the improved performance of the models in terms of accuracy and F1‐score when using WAL is demonstrated.