Twister2 Cross‐platform resource scheduler for big data. (27th July 2021)

• Record Type: Journal Article
• Authors: Uyar, Ahmet; Gunduz, Gurhan; Kamburugamuve, Supun; Wickramasinghe, Pulasthi; Widanage, Chathura; Govindarajan, Kannan; Perera, Niranda; Abeykoon, Vibhatha; Akkas, Selahattin; Fox, Geoffrey
• Other Names: Topcu Ahmet E. guestEditor.; Cibikdiken Ali Osman guestEditor.
• Is Part Of: Concurrency and computation. Volume 34:Number 9(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-07-27
• Subjects: big‐data -- HPC -- Kubernetes -- OpenMPI -- resource scheduling -- Slurm; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6502
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: Twister2 is an open‐source big data hosting environment designed to process both batch and streaming data at scale. Twister2 runs jobs in both high‐performance computing (HPC) and big data clusters. It provides a cross‐platform resource scheduler to run jobs in diverse environments. Twister2 is designed with a layered architecture to support various clusters and big data problems. In this paper, we present the cross‐platform resource scheduler of Twister2. We identify required services and explain implementation details. We present job startup delays for single jobs and multiple concurrent jobs in Kubernetes and OpenMPI clusters. We compare job startup delays for Twister2 and Spark at a Kubernetes cluster. In addition, we compare the performance of terasort algorithm on Kubernetes and bare metal clusters at AWS cloud.