Analysis and evaluation of MapReduce solutions on an HPC cluster. (February 2016)

• Record Type: Journal Article
• Authors: Veiga, Jorge; Expósito, Roberto R.; Taboada, Guillermo L.; Touriño, Juan
• Is Part Of: Computers & electrical engineering. Volume 50(2016)
• Page Start: 200
• Page End: 216
• Publication Date: 2016-02
• Subjects: MapReduce -- High Performance Computing (HPC) -- Big Data -- Energy efficiency -- InfiniBand -- Solid State Drive (SSD); Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2015.11.021
• Languages: English
• ISSNs: 0045-7906

Abstract

Highlights: Analysis and evaluation of several HPC-oriented MapReduce frameworks on a cluster. Proposal of a taxonomy to classify these frameworks according to their characteristics. Experimental configuration using several workloads, cluster sizes, networks and disk technologies. Evaluation in terms of performance and energy efficiency. Results useful to select a suitable MapReduce framework and to identify desirable characteristics for future ones. Graphical abstract: Abstract: The ever growing needs of Big Data applications are demanding challenging capabilities which cannot be handled easily by traditional systems, and thus more and more organizations are adopting High Performance Computing (HPC) to improve scalability and efficiency. Moreover, Big Data frameworks like Hadoop need to be adapted to leverage the available resources in HPC environments. This situation has caused the emergence of several HPC-oriented MapReduce frameworks, which benefit from different technologies traditionally oriented to supercomputing, such as high-performance interconnects or the message-passing interface. This work aims to establish a taxonomy of these frameworks together with a thorough evaluation, which has been carried out in terms of performance and energy efficiency metrics. Furthermore, the adaptability to emerging disks technologies, such as solid state drives, has been assessed. The results have shown that new frameworks like DataMPI can outperform Hadoop, although using IP over InfiniBand also provides significant benefits without code modifications.