A big data driven distributed density based hesitant fuzzy clustering using Apache spark with application to gene expression microarray. (March 2019) - Hosseini, Behrooz; Kiani, Kourosh
Abstract: This paper introduces a distributed density based clustering approach that benefits from the hesitant fuzzy weighted correlation coefficient as its similarity measure. All the proposed clustering steps adapt completely with the distributed Spark computation framework and the data dependency reaches its least possible using the density reachability concept. By proposing a Resilient Distributed Dataset (RDD) localized subclustering method, disk I/O burden of the MapReduce based clustering approaches has been solved as well. The comparison of the clustering results with similar works shows the superiority of the proposed algorithm in precision and cluster validity indexes. The proposed method also shows relative robustness to the presence of noise in comparison with similar recent works. The proposed approach shows better precision and validity index in comparison with MapReduce base algorithms while outperforming MapReduce based approaches in computational burden. The scalability of the proposed method is almost similar to the standard Spark machine learning library. Highlights: A new hesitant fuzzy weighted similarity measurement for gene expression. A novel density based soft clustering on the basis of Apache Spark computational model. Flexible and robust to intrinsic noise with reasonable clustering results. Parallelism with least serial bottleneck completely adapting to Spark framework. A scalable method which is efficient and suitable for processing huge volumes of big data.