MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction. Issue 6 (5th May 2021)

• Record Type: Journal Article
• Authors: Zhang, Xiao-Chen; Wu, Cheng-Kun; Yang, Zhi-Jiang; Wu, Zhen-Xing; Yi, Jia-Cai; Hsieh, Chang-Yu; Hou, Ting-Jun; Cao, Dong-Sheng
• Is Part Of: Briefings in bioinformatics. Volume 22:Issue 6(2021)
• Publication Date: 2021-05-05
• Subjects: molecular property prediction -- molecular graph BERT -- atomic representation -- deep learning -- self-supervised learning; Genetics -- Data processing -- Periodicals; Molecular biology -- Data processing -- Periodicals; Genomes -- Data processing -- Periodicals; 572.80285
• DOI: 10.1093/bib/bbab152
• Languages: English
• ISSNs: 1467-5463

Abstract

Abstract: Motivation: Accurate and efficient prediction of molecular properties is one of the fundamental issues in drug design and discovery pipelines. Traditional feature engineering-based approaches require extensive expertise in the feature design and selection process. With the development of artificial intelligence (AI) technologies, data-driven methods exhibit unparalleled advantages over the feature engineering-based methods in various domains. Nevertheless, when applied to molecular property prediction, AI models usually suffer from the scarcity of labeled data and show poor generalization ability. Results: In this study, we proposed molecular graph BERT (MG-BERT), which integrates the local message passing mechanism of graph neural networks (GNNs) into the powerful BERT model to facilitate learning from molecular graphs. Furthermore, an effective self-supervised learning strategy named masked atoms prediction was proposed to pretrain the MG-BERT model on a large amount of unlabeled data to mine context information in molecules. We found the MG-BERT model can generate context-sensitive atomic representations after pretraining and transfer the learned knowledge to the prediction of a variety of molecular properties. The experimental results show that the pretrained MG-BERT model with a little extra fine-tuning can consistently outperform the state-of-the-art methods on all 11 ADMET datasets. Moreover, the MG-BERT model leverages attention mechanisms to focus on atomic features essential to the target property, providing excellent interpretability for the trained model. The MG-BERT model does not require any hand-crafted feature as input and is more reliable due to its excellent interpretability, providing a novel framework to develop state-of-the-art models for a wide range of drug discovery tasks. Graphical abstract: