A rack-aware scalable resource management system for Hadoop YARN. (28th September 2020)

• Record Type: Journal Article
• Authors: Moses, Timothy; Inyiama, Hyacinth C.; Anigbogu, Sylvanus O.
• Is Part Of: International journal of high performance computing and networking. Volume 16:Number 1(2020)
• Page Start: 1
• Page End: 13
• Publication Date: 2020-09-28
• Subjects: MapReduce -- Hadoop -- framework -- scalable -- rack-aware -- resource manager -- big data -- rack unit resource manager; High performance computing -- Periodicals; Computer networks -- Periodicals; High performance computing; Periodicals; 004.05
• Languages: English
• ISSNs: 1740-0562

Abstract

Big data have brought in an era of data exploration and utilisation with MapReduce computational paradigm as its major enabler. Though great efforts through the implementation of Hadoop have made computation scale to tens of thousands of commodity cluster processors, the centralised architecture of resource manager has adversely affected response time in large data centres. The developed model decouples the responsibilities of resource manager by providing another layer where each daemon called rack unit resource manager (RU_RM) carries out the responsibility of allocating resources to compute nodes within its local rack to ensure low latency for large files. The application was developed and tested with Hadoop workload benchmarks used for analysis. Two performance evaluation metrics (efficiency and average task-delay ratio) were used for comparison. Efficiency quantifies average cluster utilisation while average task-delay ratio measures average delay time. Results obtained showed that as file size increases, the developed model outperforms the existing framework.