Graph partition–based data and task co‐scheduling of scientific workflow in geo‐distributed datacenters. (22nd April 2019)

• Record Type: Journal Article
• Authors: Zhang, Jinghui; Chen, Jian; Zhan, Jun; Jin, Jiahui; Song, Aibo
• Other Names: Li Gang guestEditor.; Batten Lynn guestEditor.; Foschini Luca guestEditor.; Kim Hyunbum guestEditor.; Dong Fang guestEditor.; Wu Chenshu guestEditor.; Gao Shangce guestEditor.
• Is Part Of: Concurrency and computation. Volume 31:Number 24(2019)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2019-04-22
• Subjects: data transfer -- graph partition -- hybrid genetic algorithm -- scientific workflow scheduling; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5245
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Most large‐scale scientific workflows take place in multiple collaborative datacenters for access to community‐wide resources, while adhering to each datacenter's non‐uniform resource limits. However, moving both initial input datasets with predetermined locations and intermediate datasets needing placement decisions across geo‐distributed datacenters hinders efficient execution of large‐scale data‐intensive scientific workflows. Thus, scientific workflow's data and task co‐scheduling deal with situations such as pre‐placed initial input datasets, placement of intermediate datasets and each datacenter's non‐uniform computation and storage constraint, while minimizing the cross‐datacenter data transfer. Since this scheduling problem is known to be NP‐hard, here, we propose a novel approach, based on the multilevel graph coarsening and uncoarsening framework, together with a specialized hybrid genetic algorithm having distinctive graph partition driven features of repair and local improvement, for scheduling data‐intensive scientific workflows in geo‐distributed datacenters and optimizing the cross‐datacenter data transfer volume. Extensive simulations, based on four real‐world workflow traces, show that our algorithm significantly reduces the overall geo‐distributed data transfer and demonstrate its effectiveness.