Improving Information Retrieval Performance on OCRed Text in the Absence of Clean Text Ground Truth. Issue 5 (September 2016)

• Record Type: Journal Article
• Authors: Ghosh, Kripabandhu; Chakraborty, Anirban; Parui, Swapan Kumar; Majumder, Prasenjit
• Is Part Of: Information processing & management. Volume 52:Issue 5(2016:Sep.)
• Page Start: 873
• Page End: 884
• Publication Date: 2016-09
• Subjects: Information Retrieval -- OCR error -- Word co-occurrence; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2016.03.006
• Languages: English
• ISSNs: 0306-4573

Abstract

Highlights: The proposed algorithm uses context information to segregate semantically related error variants from the unrelated ones. String similarity measures are used to join error variants with the correct query word. The algorithm is tested on Bangla, Hindi and English datasets to show that the proposed approach is language-independent. The Bangla and Hindi datasets have the clean, error-free versions for comparison. So, we have used the performances on the clean text versions as the performance upper-bounds. In addition, we have compared our method with an error modelling approach which, unlike our method, uses the clean version. The English dataset is a genuine use case scenario for our algorithm as this dataset does not have the error-free version. Our proposed method produces significant improvements on most of the baselines. We have also tested our proposed algorithm on TREC 5 Confusion track dataset and showed that our proposed method is significantly better than the baselines. Abstract: OCR errors in text harm information retrieval performance. Much research has been reported on modelling and correction of Optical Character Recognition (OCR) errors. Most of the prior work employ language dependent resources or training texts in studying the nature of errors. However, not much research has been reported that focuses on improving retrieval performance from erroneous text in the absence of training data. We propose a novel approach for detecting OCR errors and improving retrieval performance from the erroneous corpus in a situation where training samples are not available to model errors. In this paper we propose a method that automatically identifies erroneous term variants in the noisy corpus, which are used for query expansion, in the absence of clean text. We employ an effective combination of contextual information and string matching techniques. Our proposed approach automatically identifies the erroneous variants of query terms and consequently leads to improvement in retrieval performance through query expansion. Our proposed approach does not use any training data or any language specific resources like thesaurus for identification of error variants. It also does not expend any knowledge about the language except that the word delimiter is blank space. We have tested our approach on erroneous Bangla (Bengali in English) and Hindi FIRE collections, and also on TREC Legal IIT CDIP and TREC 5 Confusion track English corpora. Our proposed approach has achieved statistically significant improvements over the state-of-the-art baselines on most of the datasets.