Twister2: Design of a big data toolkit. (6th March 2019)
- Record Type:
- Journal Article
- Title:
- Twister2: Design of a big data toolkit. (6th March 2019)
- Main Title:
- Twister2: Design of a big data toolkit
- Authors:
- Kamburugamuve, Supun
Govindarajan, Kannan
Wickramasinghe, Pulasthi
Abeykoon, Vibhatha
Fox, Geoffrey - Other Names:
- Jin Hai guestEditor.
Shen Xipeng guestEditor.
Lovas Robert guestEditor.
Liao Xiaofei guestEditor.
Skjellum Anthony guestEditor.
Bangalore Purushotham V. guestEditor.
Grant Ryan E. guestEditor. - Abstract:
- Summary: Data‐driven applications are essential to handle the ever‐increasing volume, velocity, and veracity of data generated by sources such as the Web and Internet of Things (IoT) devices. Simultaneously, an event‐driven computational paradigm is emerging as the core of modern systems designed for database queries, data analytics, and on‐demand applications. Modern big data processing runtimes and asynchronous many task (AMT) systems from high performance computing (HPC) community have adopted dataflow event‐driven model. The services are increasingly moving to an event‐driven model in the form of Function as a Service (FaaS) to compose services. An event‐driven runtime designed for data processing consists of well‐understood components such as communication, scheduling, and fault tolerance. Different design choices adopted by these components determine the type of applications a system can support efficiently. We find that modern systems are limited to specific sets of applications because they have been designed with fixed choices that cannot be changed easily. In this paper, we present a loosely coupled component‐based design of a big data toolkit where each component can have different implementations to support various applications. Such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to HPC environments.
- Is Part Of:
- Concurrency and computation. Volume 32:Number 3(2020)
- Journal:
- Concurrency and computation
- Issue:
- Volume 32:Number 3(2020)
- Issue Display:
- Volume 32, Issue 3 (2020)
- Year:
- 2020
- Volume:
- 32
- Issue:
- 3
- Issue Sort Value:
- 2020-0032-0003-0000
- Page Start:
- n/a
- Page End:
- n/a
- Publication Date:
- 2019-03-06
- Subjects:
- big data -- dataflow -- event‐driven computing -- high performance computing
Parallel processing (Electronic computers) -- Periodicals
Parallel computers -- Periodicals
004.35 - Journal URLs:
- http://onlinelibrary.wiley.com/ ↗
- DOI:
- 10.1002/cpe.5189 ↗
- Languages:
- English
- ISSNs:
- 1532-0626
- Deposit Type:
- Legaldeposit
- View Content:
- Available online (eLD content is only available in our Reading Rooms) ↗
- Physical Locations:
- British Library DSC - 3405.622000
British Library DSC - BLDSS-3PM
British Library STI - ELD Digital store - Ingest File:
- 12605.xml