Neural embedding-based specificity metrics for pre-retrieval query performance prediction. Issue 4 (July 2020)

• Record Type: Journal Article
• Authors: Arabzadeh, Negar; Zarrinkalam, Fattane; Jovanovic, Jelena; Al-Obeidat, Feras; Bagheri, Ebrahim
• Is Part Of: Information processing & management. Volume 57:Issue 4(2020:Jul.)
• Publication Date: 2020-07
• Subjects: Performance prediction -- Neural embeddings -- Ad hoc retrieval; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2020.102248
• Languages: English
• ISSNs: 0306-4573

Abstract

Highlights: We show how specificity can be measured in the context of neural embeddings. We employ neural embedding-based specificity for performing query performance prediction. We publicly release two gold standard test collections for evaluating of term specificity metrics. Abstract: In information retrieval, the task of query performance prediction (QPP) is concerned with determining in advance the performance of a given query within the context of a retrieval model. QPP has an important role in ensuring proper handling of queries with varying levels of difficulty. Based on the extant literature, query specificity is an important indicator of query performance and is typically estimated using corpus-specific frequency-based specificity metrics However, such metrics do not consider term semantics and inter-term associations. Our work presented in this paper distinguishes itself by proposing a host of corpus-independent specificity metrics that are based on pre-trained neural embeddings and leverage geometric relations between terms in the embedding space in order to capture the semantics of terms and their interdependencies. Specifically, we propose three classes of specificity metrics based on pre-trained neural embeddings: neighborhood-based, graph-based, and cluster-based metrics. Through two extensive and complementary sets of experiments, we show that the proposed specificity metrics (1) are suitable specificity indicators, based on the gold standards derived from knowledge hierarchies (Wikipedia category hierarchy and DMOZ taxonomy), and (2) have better or competitive performance compared to the state of the art QPP metrics, based on both TREC ad hoc collections namely Robust'04, Gov2 and ClueWeb'09 and ANTIQUE question answering collection. The proposed graph-based specificity metrics, especially those that capture a larger number of inter-term associations, proved to be the most effective in both query specificity estimation and QPP. We have also publicly released two test collections (i.e. specificity gold standards) that we built from the Wikipedia and DMOZ knowledge hierarchies.