Cluster analysis via random partition distributions. (12th November 2022)

• Record Type: Journal Article
• Authors: Dahl, David B.; Andros, Jacob; Carter, J. Brandon
• Is Part Of: Statistical analysis and data mining. Volume 16:Number 2(2023)
• Page Start: 135
• Page End: 148
• Publication Date: 2022-11-12
• Subjects: dendrogram -- Ewens–Pitman attraction distribution -- hierarchical clustering -- k‐medoids clustering -- random partition models; Data mining -- Statistical methods -- Periodicals; 006.312
• DOI: 10.1002/sam.11602
• Languages: English
• ISSNs: 1932-1864

Abstract

Abstract: Hierarchical and k‐medoids clustering are deterministic clustering algorithms defined on pairwise distances. We use these same pairwise distances in a novel stochastic clustering procedure based on a probability distribution. We call our proposed method CaviarPD, a portmanteau from cluster analysis via random partition distributions. CaviarPD first samples clusterings from a distribution on partitions and then finds the best cluster estimate based on these samples using algorithms to minimize an expected loss. Using eight case studies, we show that our approach produces results as close to the truth as hierarchical and k‐medoids methods, and has the additional advantage of allowing for a probabilistic framework to assess clustering uncertainty. The method provides an intuitive graphical representation of clustering uncertainty through pairwise probabilities from partition samples. A software implementation of the method is available in the CaviarPD package for R.