Query-based summarization of discussion threads. (16th April 2019)

• Record Type: Journal Article
• Authors: Verberne, Suzan; Krahmer, Emiel; Wubben, Sander; van den Bosch, Antal
• Is Part Of: Natural language engineering. Volume 26:Part 1(2020)
• Page Start: 3
• Page End: 29
• Publication Date: 2019-04-16
• Subjects: query-based summarization, -- discussion forums, -- reference summaries, -- word embeddings, -- evaluation; Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324919000123
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: In this paper, we address query-based summarization of discussion threads. New users can profit from the information shared in the forum, Please check if the inserted city and country names in the affiliations are correct. if they can find back the previously posted information. However, discussion threads on a single topic can easily comprise dozens or hundreds of individual posts. Our aim is to summarize forum threads given real web search queries. We created a data set with search queries from a discussion forum's search engine log and the discussion threads that were clicked by the user who entered the query. For 120 thread–query combinations, a reference summary was made by five different human raters. We compared two methods for automatic summarization of the threads: a query-independent method based on post features, and Maximum Marginal Relevance (MMR), a method that takes the query into account. We also compared four different word embeddings representations as alternative for standard word vectors in extractive summarization. We find (1) that the agreement between human summarizers does not improve when a query is provided that: (2) the query-independent post features as well as a centroid-based baseline outperform MMR by a large margin; (3) combining the post features with query similarity gives a small improvement over the use of post features alone; and (4) for the word embeddings, a match in domain appears to be more important than corpus size and dimensionality. However, the differences between the models were not reflected by differences in quality of the summaries created with help of these models. We conclude that query-based summarization with web queries is challenging because the queries are short, and a click on a result is not a direct indicator for the relevance of the result.