A new computationally efficient algorithm for record linkage with field dependency and missing data imputation. (January 2018)

• Record Type: Journal Article
• Authors: Ferguson, John; Hannigan, Ailish; Stack, Austin
• Is Part Of: International journal of medical informatics. Volume 109(2018)
• Page Start: 70
• Page End: 75
• Publication Date: 2018-01
• Subjects: Record linkage -- Fellegi/Sunter -- Conditional independence -- EM-algorithm -- Log-linear models; Medical informatics -- Periodicals; Information science -- Periodicals; Computers -- Periodicals; Medical technology -- Periodicals; Medical Informatics -- Periodicals; Technology, Medical -- Periodicals; Computers; Information science; Medical informatics; Medical technology; Electronic journals; Periodicals; Electronic journals; 610.285
• DOI: 10.1016/j.ijmedinf.2017.10.021
• Languages: English
• ISSNs: 1386-5056

Abstract

Highlights: Record linkage algorithms aim to identify pairs of records in two or more databases, that correspond to the same individual. Here, we propose a novel record linkage algorithm that incorporates missing data and correlation between field agreement indicators. We demonstrate that our algorithm can achieve more accurate record linkage, compared to the original probabilistic linkage algorithm of Fellegi and Sunter. Our algorithm is computationally efficient, and can be used to link large databases consisting of millions of records. An R-package, corlink, has been developed to implement the new algorithm and can be downloaded from the CRAN repository. Abstract: Record linkage algorithms aim to identify pairs of records that correspond to the same individual from two or more datasets. In general, fields that are common to both datasets are compared to determine which record-pairs to link. The classic model for probabilistic linkage was proposed by Fellegi and Sunter and assumes that individual fields common to both datasets are completely observed, and that the field agreement indicators are conditionally independent within the subsets of record pairs corresponding to the same and differing individuals. Herein, we propose a novel record linkage algorithm that is independent of these two baseline assumptions. We demonstrate improved performance of the algorithm in the presence of missing data and correlation patterns between the agreement indicators. The algorithm is computationally efficient and can be used to link large databases consisting of millions of record pairs. An R-package, corlink, has been developed to implement the new algorithm and can be downloaded from the CRAN repository.