Deep contextualized embeddings for quantifying the informative content in biomedical text summarization. (February 2020)

• Record Type: Journal Article
• Authors: Moradi, Milad; Dorffner, Georg; Samwald, Matthias
• Is Part Of: Computer methods and programs in biomedicine. Volume 184(2020)
• Publication Date: 2020-02
• Subjects: Biomedical text mining -- Text summarization -- Contextualized embeddings -- Deep learning, domain knowledge -- Clustering; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2019.105117
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: A deep bidirectional language model is used to capture the context of sentences. The shared content between sentences is quantified using contextualized embeddings. A hierarchical clustering algorithm is utilized to identify the most relevant sentences. The summarizer improves the performance of biomedical text summarization. Contextualized embeddings can effectively capture the context in biomedical summarization. Abstract: Background and Objective: Capturing the context of text is a challenging task in biomedical text summarization. The objective of this research is to show how contextualized embeddings produced by a deep bidirectional language model can be utilized to quantify the informative content of sentences in biomedical text summarization. Methods: We propose a novel summarization method that utilizes contextualized embeddings generated by the Bidirectional Encoder Representations from Transformers (BERT) model, a deep learning model that recently demonstrated state-of-the-art results in several natural language processing tasks. We combine different versions of BERT with a clustering method to identify the most relevant and informative sentences of input documents. Using the ROUGE toolkit, we evaluate the summarizer against several methods previously described in literature. Results: The summarizer obtains state-of-the-art results and significantly improves the performance of biomedical text summarization in comparison to a set of domain-specific and domain-independent methods. The largest language model not specifically pretrained on biomedical text outperformed other models. However, among language models of the same size, the one further pretrained on biomedical text obtained best results. Conclusions: We demonstrate that a hybrid system combining a deep bidirectional language model and a clustering method yields state-of-the-art results without requiring labor-intensive creation of annotated features or knowledge bases or computationally demanding domain-specific pretraining. This study provides a starting point towards investigating deep contextualized language models for biomedical text summarization.