Text classification using improved bidirectional transformer. (18th July 2021)

• Record Type: Journal Article
• Authors: Tezgider, Murat; Yildiz, Beytullah; Aydin, Galip
• Other Names: Topcu Ahmet E. guestEditor.; Cibikdiken Ali Osman guestEditor.
• Is Part Of: Concurrency and computation. Volume 34:Number 9(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-07-18
• Subjects: attention -- deep learning -- machine learning -- text classification -- text processing -- transformer; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6486
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: Text data have an important place in our daily life. A huge amount of text data is generated everyday. As a result, automation becomes necessary to handle these large text data. Recently, we are witnessing important developments with the adaptation of new approaches in text processing. Attention mechanisms and transformers are emerging as methods with significant potential for text processing. In this study, we introduced a bidirectional transformer (BiTransformer) constructed using two transformer encoder blocks that utilize bidirectional position encoding to take into account the forward and backward position information of text data. We also created models to evaluate the contribution of attention mechanisms to the classification process. Four models, including long short term memory, attention, transformer, and BiTransformer, were used to conduct experiments on a large Turkish text dataset consisting of 30 categories. The effect of using pretrained embedding on models was also investigated. Experimental results show that the classification models using transformer and attention give promising results compared with classical deep learning models. We observed that the BiTransformer we proposed showed superior performance in text classification.