Twister2: Design of a big data toolkit. (6th March 2019)

• Record Type: Journal Article
• Authors: Kamburugamuve, Supun; Govindarajan, Kannan; Wickramasinghe, Pulasthi; Abeykoon, Vibhatha; Fox, Geoffrey
• Other Names: Jin Hai guestEditor.; Shen Xipeng guestEditor.; Lovas Robert guestEditor.; Liao Xiaofei guestEditor.; Skjellum Anthony guestEditor.; Bangalore Purushotham V. guestEditor.; Grant Ryan E. guestEditor.
• Is Part Of: Concurrency and computation. Volume 32:Number 3(2020)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2019-03-06
• Subjects: big data -- dataflow -- event‐driven computing -- high performance computing; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5189
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Data‐driven applications are essential to handle the ever‐increasing volume, velocity, and veracity of data generated by sources such as the Web and Internet of Things (IoT) devices. Simultaneously, an event‐driven computational paradigm is emerging as the core of modern systems designed for database queries, data analytics, and on‐demand applications. Modern big data processing runtimes and asynchronous many task (AMT) systems from high performance computing (HPC) community have adopted dataflow event‐driven model. The services are increasingly moving to an event‐driven model in the form of Function as a Service (FaaS) to compose services. An event‐driven runtime designed for data processing consists of well‐understood components such as communication, scheduling, and fault tolerance. Different design choices adopted by these components determine the type of applications a system can support efficiently. We find that modern systems are limited to specific sets of applications because they have been designed with fixed choices that cannot be changed easily. In this paper, we present a loosely coupled component‐based design of a big data toolkit where each component can have different implementations to support various applications. Such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to HPC environments.