A new technique of selecting an optimal blocking method for better record linkage. (September 2018)

• Record Type: Journal Article
• Authors: O'Hare, Kevin; Jurek, Anna; de Campos, Cassio
• Is Part Of: Information systems. Volume 77(2018)
• Page Start: 151
• Page End: 166
• Publication Date: 2018-09
• Subjects: Record linkage -- Blocking -- Entity resolution; Database management -- Periodicals; Electronic data processing -- Periodicals; Bases de données -- Gestion -- Périodiques; Informatique -- Périodiques; Database management; Electronic data processing; Periodicals; 005.7
• DOI: 10.1016/j.is.2018.06.006
• Languages: English
• ISSNs: 0306-4379

Abstract

Highlights: Current evaluation metrics fail to select optimal blocking methods for record linkage. Proposed evaluation technique allows for better comparison between blocking methods. Proposed selection technique selects top performing blocking methods for new datasets. Experimental evaluations demonstrate the effectiveness of these novel techniques. Abstract: Record linkage, referred to also as entity resolution, is the process of identifying pairs of records representing the same real world entity (e.g. a person) within a dataset or across multiple datasets. In order to reduce the number of record comparisons, record linkage frameworks initially perform a process referred to as blocking, which involves splitting records into a set of blocks using a partition (or blocking) scheme. This restricts comparisons among records that belong to the same block during the linkage process. Existing blocking methods are often evaluated using different metrics and independently of the choice of the subsequent linkage method, which makes the choice of an optimal approach very subjective. In this paper we demonstrate that existing evaluation metrics fail to provide strong evidence to support the selection of an optimal blocking method. We conduct an extensive evaluation of different blocking methods using multiple datasets and some commonly applied linkage techniques to show that evaluation of a blocking method must take into consideration the subsequent linkage phase. We propose a novel evaluation technique that takes into consideration multiple factors including the end-to-end running time of the combined blocking and linkage phases as well as the linkage technique used. We empirically demonstrate using multiple datasets that according to this novel evaluation technique some blocking methods can be fairly considered superior to others, while some should be deemed incomparable according to those factors. Finally, we propose a novel blocking method selection procedure that takes into consideration the linkage proficiency and end-to-end time of different blocking methods combined with a given linkage technique. We show that this technique is able to select the best or near best blocking method for unseen data.