A partition based method for finding highly correlated pairs. (1st October 2010)

• Record Type: Journal Article
• Authors: Li, Shuxin; Lang, Sheau-Dong
• Is Part Of: International journal of data mining, modelling and management. Volume 2:Number 4(2010)
• Page Start: 334
• Page End: 350
• Publication Date: 2010-10-01
• Subjects: correlation -- association rules -- Pearson correlation coefficients -- transactional databases -- data mining -- partition -- highly correlated pairs; Data mining -- Periodicals; Information science -- Periodicals; Databases -- Periodicals; 005.7
• Languages: English
• ISSNs: 1759-1163

Abstract

The problem of finding highly correlated pairs is to output all item pairs whose (Pearson) correlation coefficients are greater than a user-specified correlation threshold. Effective discovery of such item pairs is of primary importance in many real data mining applications. Algorithm and Taper algorithm are special cases of our new algorithm with respect to the number of segments. Experimental results on real datasets demonstrate the feasibility and superiority of our algorithm. Recently, the Taper algorithm is developed to discover the set of highly correlated item pairs. In this paper, we present a generalised Taper algorithm to find strongly correlated pairs between items by partitioning the collection of transactions into different segments, so as to achieve better pruning effect and less running time. Consequently, it can be proved that both are naive.