MapReduce service provisioning for frequent big data jobs on clouds considering data transfers. (October 2018)

• Record Type: Journal Article
• Authors: Nabavinejad, Seyed Morteza; Goudarzi, Maziar; Abedi, Saeed
• Is Part Of: Computers & electrical engineering. Volume 71(2018)
• Page Start: 594
• Page End: 610
• Publication Date: 2018-10
• Subjects: Big data -- MapReduce -- Cloud computing -- Hadoop -- Energy efficiency; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2018.08.005
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Many companies regularly run Big Data analysis, and need to optimize their resource usage considering cost, deadline, and environmental impact simultaneously. The cloud allows choosing from various virtual machines (VM) where the number and type of VMs affect the outcome such as the time for data placement and data shuffle phases, a task's energy consumption and execution time, and the makespan of jobs. We provide provisioning and scheduling algorithms to minimize environmental impact, considering the above factors, for frequently executed MapReduce jobs. To mathematically model the problem and obtain the optimal solution, we present an Integer Linear Programming (ILP) model and then continue with two heuristic algorithms. We compare proposed algorithms against a number of rivals using extensive simulations based on publicly available real-world data. The results demonstrate that our algorithms can achieve near-optimal solutions, e.g., sometime even within 0.39% of the optimal solution obtained by ILP regarding energy consumption.