Evaluation of text summaries without human references based on the linear optimization of content metrics using a genetic algorithm. (1st April 2021)

• Record Type: Journal Article
• Authors: Rojas-Simón, Jonathan; Ledeneva, Yulia; García-Hernández, René Arnulfo
• Is Part Of: Expert systems with applications. Volume 167(2021)
• Publication Date: 2021-04-01
• Subjects: Evaluation of text summaries -- Linear optimization of content metrics -- Genetic algorithm -- ROUGE-C -- Latent semantic analysis -- Jensen-Shannon divergence; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2020.113827
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: The proposed evaluation provides a better correlation than state-of-the-art methods. 31 state-of-the-art metrics are combined to generate an optimized evaluation metric. The proposed evaluation method enables a balanced correlation improvement. The relevance of evaluation metrics presents a direct relation of individual correlation. State-of-the-art metrics with higher correlations improve the resultant final rank. Abstract: The Evaluation of Text Summaries (ETS) has been a task of constant challenges to the development of Automatic Text Summarization (ATS). Within the ATS task, the ETS is crucial to determine the performance of text summaries. Over the last two decades, the scientific community has used the ROUGE system as a standard package to assess the content of automatic summaries. However, if there are not human-made summaries (called human references), then the evaluation cannot be carried out. For this reason, the different state-of-the-art evaluation methods have been proposed that analyze the summary content using the source documents. Nonetheless, these methods do not highly correlate with human assessment. In this paper, a linear optimization of content-based metrics is proposed using a Genetic Algorithm (GA) to improve the correlation between automatic and manual evaluation. The proposed method combines 31 content metrics based on the evaluation without human references. The results of the linear optimization show correlation improvements concerning other evaluation metrics on DUC01 and DUC02 datasets.