Towards effective scheduling policies for many‐task applications: Practice and experience based on HTCaaS. (24th August 2017) - Kim, Jik‐Soo; Quang, Bui; Rho, Seungwoo; Kim, Seoyoung; Kim, Sangwan; Breton, Vincent; Hwang, Soonwook
Summary: In this paper, we conduct a comparative study of relatively simple yet effective scheduling policies for many‐task applications where multiple users with varying numbers of tasks are actively sharing a common system infrastructure. We have implemented three different scheduling mechanisms that can address fairness and user response time respectively in a common middleware stack called HTCaaS, which is a pilot‐job–based multi‐level scheduling system running on top of production‐level clusters. As a representative case of our many‐task applications, we have leveraged the virtual screening application, which is a computational technique used in drug discovery process to select the most promising candidate drugs for in vitro testing from millions of chemical compounds, which typically requires a substantial amount of computing resources and efficient processing of docking simulations. Our comparative experimental results of different scheduling policies show how we can effectively support multiple users in a shared resource environment by balancing between user satisfaction and overall system performance, provide guidelines to improve system utilization, and address additional technical challenges to support various many‐task applications.