Towards effective scheduling policies for many‐task applications: Practice and experience based on HTCaaS. (24th August 2017)

• Record Type: Journal Article
• Authors: Kim, Jik‐Soo; Quang, Bui; Rho, Seungwoo; Kim, Seoyoung; Kim, Sangwan; Breton, Vincent; Hwang, Soonwook
• Is Part Of: Concurrency and computation. Volume 29:Number 21(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2017-08-24
• Subjects: HTCaaS -- many‐task computing -- multi‐level scheduling -- resource allocation -- virtual screening; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.4242
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: In this paper, we conduct a comparative study of relatively simple yet effective scheduling policies for many‐task applications where multiple users with varying numbers of tasks are actively sharing a common system infrastructure. We have implemented three different scheduling mechanisms that can address fairness and user response time respectively in a common middleware stack called HTCaaS, which is a pilot‐job–based multi‐level scheduling system running on top of production‐level clusters. As a representative case of our many‐task applications, we have leveraged the virtual screening application, which is a computational technique used in drug discovery process to select the most promising candidate drugs for in vitro testing from millions of chemical compounds, which typically requires a substantial amount of computing resources and efficient processing of docking simulations. Our comparative experimental results of different scheduling policies show how we can effectively support multiple users in a shared resource environment by balancing between user satisfaction and overall system performance, provide guidelines to improve system utilization, and address additional technical challenges to support various many‐task applications.