Verbosity normalized pseudo-relevance feedback in information retrieval. Issue 2 (March 2018)

• Record Type: Journal Article
• Authors: Na, Seung-Hoon; Kim, Kangil
• Is Part Of: Information processing & management. Volume 54:Issue 2(2018:Mar.)
• Page Start: 219
• Page End: 239
• Publication Date: 2018-03
• Subjects: Pseudo-relevance feedback -- Verbosity normalization -- Scope normalization -- Term frequency; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2017.09.006
• Languages: English
• ISSNs: 0306-4573

Abstract

Highlights: We examine document length normalization on the pseudo-relevance feedback setting. We assume verbosity effect on term weights and scope effect on term selection. We generalize the existing two-stage normalization for the pseudo-relevance feedback. We apply the generalized two-stage normalization to the latent concept expansion. The resulting model improves the latent concept expansion on standard TREC datasets. Abstract: Document length normalization is one of the fundamental components in a retrieval model because term frequencies can readily be increased in long documents. The key hypotheses in literature regarding document length normalization are the verbosity and scope hypotheses, which imply that document length normalization should consider the distinguishing effects of verbosity and scope on term frequencies. In this article, we extend these hypotheses in a pseudo-relevance feedback setting by assuming the verbosity hypothesis on the feedback query model, which states that the verbosity of an expanded query should not be high. Furthermore, we postulate the following two effects of document verbosity on a feedback query model that easily and typically holds in modern pseudo-relevance feedback methods: 1) the verbosity-preserving effect : the query verbosity of a feedback query model is determined by feedback document verbosities; 2) the verbosity-sensitive effect : highly verbose documents more significantly and unfairly affect the resulting query model than normal documents do. By considering these effects, we propose verbosity normalized pseudo-relevance feedback, which is straightforwardly obtained by replacing original term frequencies with their verbosity-normalized term frequencies in the pseudo-relevance feedback method. The results of the experiments performed on three standard TREC collections show that the proposed verbosity normalized pseudo-relevance feedback consistently provides statistically significant improvements over conventional methods, under the settings of the relevance model and latent concept expansion .