Explainability in transformer models for functional genomics. Issue 5 (8th April 2021)

• Record Type: Journal Article
• Authors: Clauwaert, Jim; Menschaert, Gerben; Waegeman, Willem
• Is Part Of: Briefings in bioinformatics. Volume 22:Issue 5(2021)
• Publication Date: 2021-04-08
• Subjects: interpretable neural networks -- transformers -- functional genomics -- DNA-binding sites; Genetics -- Data processing -- Periodicals; Molecular biology -- Data processing -- Periodicals; Genomes -- Data processing -- Periodicals; 572.80285
• DOI: 10.1093/bib/bbab060
• Languages: English
• ISSNs: 1467-5463

Abstract

Abstract: The effectiveness of deep learning methods can be largely attributed to the automated extraction of relevant features from raw data. In the field of functional genomics, this generally concerns the automatic selection of relevant nucleotide motifs from DNA sequences. To benefit from automated learning methods, new strategies are required that unveil the decision-making process of trained models. In this paper, we present a new approach that has been successful in gathering insights on the transcription process in Escherichia coli . This work builds upon a transformer-based neural network framework designed for prokaryotic genome annotation purposes. We find that the majority of subunits (attention heads) of the model are specialized towards identifying transcription factors and are able to successfully characterize both their binding sites and consensus sequences, uncovering both well-known and potentially novel elements involved in the initiation of the transcription process. With the specialization of the attention heads occurring automatically, we believe transformer models to be of high interest towards the creation of explainable neural networks in this field.