High‐performance iterative dataflow abstractions in Twister2:TSet. (14th September 2020)

• Record Type: Journal Article
• Authors: Wickramasinghe, Pulasthi; Perera, Niranda; Kamburugamuve, Supun; Govindarajan, Kannan; Abeykoon, Vibhatha; Widanage, Chathura; Uyar, Ahmet; Gunduz, Gurhan; Akkas, Selahattin; Fox, Geoffrey
• Is Part Of: Concurrency and computation. Volume 34:Number 12(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-09-14
• Subjects: batch -- big data -- dataflow -- iterative -- mapreduce -- stream; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5998
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: The dataflow model is gradually becoming the de facto standard for big data applications. While many popular frameworks are built around this model, very little research has been done on understanding its inner workings, which in turn has led to inefficiencies in existing frameworks. It is important to note that understanding the relationship between dataflow and high performance computing (HPC) building blocks allows us to address and alleviate many of these fundamental inefficiencies by learning from the extensive research literature in the HPC community. In this article, we present TSets, the dataflow abstraction of Twister2, which is a big data framework designed for high‐performance dataflow and iterative computations. We discuss the dataflow model adopted by TSets and the rationale behind implementing iteration handling at the worker level. Finally, we evaluate TSets to show the performance of the framework and the importance of the worker level iteration model.