Relevance theory and distributions of judgments in document retrieval. Issue 5 (September 2017)

• Record Type: Journal Article
• Authors: White, Howard D.
• Is Part Of: Information processing & management. Volume 53:Issue 5(2017:Sep.)
• Page Start: 1080
• Page End: 1102
• Publication Date: 2017-09
• Subjects: Scale language -- Cognitive effects -- Processing effort; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2017.02.010
• Languages: English
• ISSNs: 0306-4573

Abstract

Highlights: Links information retrieval evaluation to Sperber and Wilson's relevance theory. Analyzes the language of relevance judgment scales in more than 50 experiments. Focuses on scale-induced effort as a variable in relevance judgments. Shows that vague scales tend to produce U- or V-shaped distributions of judgments. Shows that strict scales tend to produce right-skewed distributions of judgments. Abstract: This article extends relevance theory (RT) from linguistic pragmatics into information retrieval. Using more than 50 retrieval experiments from the literature as examples, it applies RT to explain the frequency distributions of documents on relevance scales with three or more points. The scale points, which judges in experiments must consider in addition to queries and documents, are communications from researchers. In RT, the relevance of a communication varies directly with its cognitive effects and inversely with the effort of processing it. Researchers define and/or label the scale points to measure the cognitive effects of documents on judges. However, they apparently assume that all scale points as presented are equally easy for judges to process. Yet the notion that points cost variable effort explains fairly well the frequency distributions of judgments across them. By hypothesis, points that cost more effort are chosen by judges less frequently. Effort varies with the vagueness or strictness of scale-point labels and definitions. It is shown that vague scales tend to produce U- or V-shaped distributions, while strict scales tend to produce right-skewed distributions. These results reinforce the paper's more general argument that RT clarifies the concept of relevance in the dialogues of retrieval evaluation.