RepSum: A general abstractive summarization framework with dynamic word embedding representation correction. (May 2023) - Feng, Jianzhou; Long, Jing; Han, Chunlong; Ren, Zhongcan; Wang, Qin
Abstract: Abstractive summarization is flexible and allows the model to generate new words and phrases. However, the familiar words are more likely to be selected as abstract candidate words in the process of abstractive summarization, causing the generated abstract to diverge from the reference. In our consideration, this is caused by representation degeneration of the pre-trained word embedding. Therefore, this paper proposes a general abstractive summarization framework with dynamic word embedding representation correction (RepSum ). The representation correction algorithm identifies the dimension most relevant to word frequency and eliminates the word frequency features. Then the distribution of word embeddings will be more even. As a result, the words will be selected as candidate words without frequency bias to improve the quality of the abstract. The experimental results illustrate that RepSum performs better than the benchmark model in summary quality, demonstrating our method's effectiveness. Graphical abstract: Highlights: An embedding representation correction algorithm based on word frequency feature elimination is proposed. The algorithm identifies the dimension most relevant to word frequency and eliminates the word frequency features. Then the distribution of word embeddings will be more even and the words will be selected as candidate words without frequency bias.