A transformer architecture based on BERT and 2D convolutional neural network to identify DNA enhancers from sequence information. Issue 5 (5th February 2021)

• Record Type: Journal Article
• Authors: Le, Nguyen Quoc Khanh; Ho, Quang-Thai; Nguyen, Trinh-Trung-Duong; Ou, Yu-Yen
• Is Part Of: Briefings in bioinformatics. Volume 22:Issue 5(2021)
• Publication Date: 2021-02-05
• Subjects: contextualized word embedding -- BERT -- convolutional neural network -- biological sequence -- DNA enhancer -- NLP transformer; Genetics -- Data processing -- Periodicals; Molecular biology -- Data processing -- Periodicals; Genomes -- Data processing -- Periodicals; 572.80285
• DOI: 10.1093/bib/bbab005
• Languages: English
• ISSNs: 1467-5463

Abstract

Abstract: Recently, language representation models have drawn a lot of attention in the natural language processing field due to their remarkable results. Among them, bidirectional encoder representations from transformers (BERT) has proven to be a simple, yet powerful language model that achieved novel state-of-the-art performance. BERT adopted the concept of contextualized word embedding to capture the semantics and context of the words in which they appeared. In this study, we present a novel technique by incorporating BERT-based multilingual model in bioinformatics to represent the information of DNA sequences. We treated DNA sequences as natural sentences and then used BERT models to transform them into fixed-length numerical matrices. As a case study, we applied our method to DNA enhancer prediction, which is a well-known and challenging problem in this field. We then observed that our BERT-based features improved more than 5–10% in terms of sensitivity, specificity, accuracy and Matthews correlation coefficient compared to the current state-of-the-art features in bioinformatics. Moreover, advanced experiments show that deep learning (as represented by 2D convolutional neural networks; CNN) holds potential in learning BERT features better than other traditional machine learning techniques. In conclusion, we suggest that BERT and 2D CNNs could open a new avenue in biological modeling using sequence information.