Using Spearman's correlation coefficients for exploratory data analysis on big dataset. (18th December 2015)

• Record Type: Journal Article
• Authors: Xiao, Chengwei; Ye, Jiaqi; Esteves, Rui Máximo; Rong, Chunming
• Other Names: Higuera‐Toledano M. Teresa guestEditor.; Brinkschulte Uwe guestEditor.; Rettberg Achim guestEditor.; Qiang Weizhong guestEditor.; Zheng Xianghan guestEditor.; Hsu Ching‐Hsien guestEditor.
• Is Part Of: Concurrency and computation. Volume 28:Number 14(2016)
• Page Start: 3866
• Page End: 3878
• Publication Date: 2015-12-18
• Subjects: exploratory data analysis -- correlation analysis -- Spearman correlation coefficient -- p‐value -- vibration analysis -- pump state; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3745
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Correlation analysis is both popular and useful in a number of social networking research, particularly in the exploratory data analysis. In this paper, three well‐known and often‐used correlation coefficients, Pearson product–moment correlation coefficient, Spearman, and Kendall rank correlation coefficients, are compared from definition to application domain. Based on the characteristics of the pump's vibration dataset, the nonparametric and distribution‐free Spearman rank correlation coefficient is introduced to analyze the relationship between the pump's working state and each of the 207′880 variables. The percentage of variables and exact variables' tables with high Spearman's correlation coefficients for states I and II, states I and III, states II and III, and three states in different files are obtained respectively, which has important valuation for the future research of the unsupervised machine learning system. Copyright © 2015 John Wiley & Sons, Ltd.