Self‐adaptation and mutual adaptation for distributed scheduling in benevolent clouds. (23rd September 2016)

• Record Type: Journal Article
• Authors: Xiao, Zheng; Liang, Pijun; Tong, Zhao; Li, Kenli; Khan, Samee U.; Li, Keqin
• Is Part Of: Concurrency and computation. Volume 29:Number 5(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2016-09-23
• Subjects: collaboration -- distributed computing -- hybrid cloud -- machine learning -- Q‐learning -- task scheduling; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3939
• Languages: English
• ISSNs: 1532-0626

Abstract

SUMMARY: Joint service involving several clouds is an emerging form of cloud computing. In hybrid clouds, the schedulers within 1 cloud must not only self‐adapt to the job arrival processes and the workload but also mutually adapt to the scheduling polices of other schedulers. However, as a combinatorial optimization problem, scheduling is challenged by the adaptation to those dynamics and uncertain behaviors of the peers. This article studies the collaboration among benevolent clouds that are cooperative in nature and willing to accept jobs from other clouds. We take advantage of machine learning and propose a distributed scheduling mechanism to learn the knowledge of job model, resource performance, and others' policies. Without explicit modeling and prediction, machine learning guides scheduling decisions based on experiences. To examine the performance of our approach, we conducted simulation using the SP2 job workload log of the San Diego Supercomputer Center under a test bed based on agent‐based systems—SWARM. The results validate that our approach has much shorter mean response time than 5 typical dynamic scheduling algorithms—opportunistic load balancing, minimum execution time, minimum completion time, switching algorithm, and k ‐percent best. A better collaboration in hybrid cloud is achieved by full adaptation.