Dispel4py: A Python framework for data-intensive scientific computing. (July 2017)

• Record Type: Journal Article
• Authors: Filguiera, Rosa; Krause, Amrey; Atkinson, Malcolm; Klampanos, Iraklis; Moreno, Alexander
• Is Part Of: International journal of high performance computing applications. Volume 31:Number 4(2017)
• Page Start: 316
• Page End: 334
• Publication Date: 2017-07
• Subjects: Data-intensive computing -- e-infrastructures -- data streaming -- scientific workflows -- programming frameworks; High performance computing -- Periodicals; Supercomputers -- Periodicals; 004.1105
• DOI: 10.1177/1094342016649766
• Languages: English
• ISSNs: 1094-3420

Abstract

This paper presentsdispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations.dispel4py enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to mapdispel4py abstract workflows optimally onto target platforms chosen dynamically. We present fourdispel4py mappings: Apache Storm, message-passing interface (MPI), multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. Thedispel4py.org is an open-source project to which we invite participation. The effective mapping ofdispel4py onto multiple target infrastructures demonstrates exploitation of data-intensive and high-performance computing (HPC) architectures and consistent scalability.