APFA: Automated product feature alignment for duplicate detection. (15th July 2021)

• Record Type: Journal Article
• Authors: Valstar, Nick; Frasincar, Flavius; Brauwers, Gianni
• Is Part Of: Expert systems with applications. Volume 174(2021)
• Publication Date: 2021-07-15
• Subjects: Duplicate detection -- Automated pre-processing -- Product comparison -- E-commerce; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2021.114759
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: We propose an automated pre-processing algorithm for product duplicate detection. The pre-processing phase employs new key metrics for feature alignment. A novel brand analyzer and title analyzer are presented. The proposed methods are tested using data from four real-world Web shops. The pre-processing phase significantly improves both effectiveness and speed. Abstract: To keep up with the growing interest of using Web shops for product comparison, we have developed a method that targets the problem of product duplicate detection. If duplicates can be discovered correctly and quickly, customers can compare products in an efficient manner. We build upon the state-of-the-art Multi-component Similarity Method (MSM) for product duplicate detection by developing an automated pre-processing phase that occurs before the similarities between products are calculated. Specifically, in this prior phase the features of products are aligned between Web shops, using metrics such as the data type, coverage, and diversity of each key, as well as the distribution and used measurement units of their corresponding values. With this information, the values of these keys can be more meaningfully and efficiently employed in the process of comparing products. Applying our method to a real-world dataset of 1629 TV's across 4 Web shops, we find that we increase the speed of the product similarity phase by roughly a factor 3 due to fewer meaningless comparisons, an improved brand analyzer, and a renewed title analyzer. Moreover, in terms of quality of duplicate detection, we significantly outperform MSM with an F 1 -measure of 0.746 versus 0.525.