A sliding window‐based dynamic load balancing for heterogeneous Hadoop clusters. (6th January 2016)

• Record Type: Journal Article
• Authors: Liu, Yang; Jing, Weizhe; Liu, Youbo; Lv, Lin; Qi, Man; Xiang, Yang; Li, Maozhen
• Is Part Of: Concurrency and computation. Volume 29:Number 3(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2016-01-06
• Subjects: mapreduce -- Hadoop -- dynamic load balancing -- big data; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3763
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: At present MapReduce computing model‐based Hadoop framework has gradually become the most famous distributed computing framework because of its remarkable features such as scalability, fault tolerance, data security, and powerful IO ability. However, Hadoop framework only supports limited load balancing policies, which may result in performance deterioration in heterogeneous clusters. Additionally Hadoop does not have advanced dynamic load balancing mechanism in enabling its optimal performance in dynamic environment. This paper presents a sliding window‐based dynamic load balancing algorithm, which specially aims at balancing the load among the heterogeneous nodes during the Hadoop job processing. The presented algorithm is evaluated in both simulated and physical environments. The experimental results show that the performances in terms of efficiency of Hadoop cluster can be significantly improved. Copyright © 2016 John Wiley & Sons, Ltd.