Submarine: A subscription‐based data streaming framework for integrating large facilities and advanced cyberinfrastructure. (11th April 2019)

• Record Type: Journal Article
• Authors: Zamani, Ali Reza; AbdelBaky, Moustafa; Balouek‐Thomert, Daniel; Villalobos, J. J.; Rodero, Ivan; Parashar, Manish
• Other Names: Wu Yulei guestEditor.; Yan Zheng guestEditor.; Zhao Zhiwei guestEditor.; Al‐Dubai Ahmed guestEditor.; Li Tonglin guestEditor.; Xie Bing guestEditor.; Zhang Boyu guestEditor.
• Is Part Of: Concurrency and computation. Volume 32:Number 16(2020)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2019-04-11
• Subjects: large‐scale scientific facilities -- ocean observatories initiative -- real‐time data delivery -- stream processing; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5256
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Large scientific facilities provide researchers with instrumentation, data, and data products that can accelerate scientific discovery. However, increasing data volumes coupled with limited local computational power prevents researchers from taking full advantage of what these facilities can offer. Many researchers looked into using commercial and academic cyberinfrastructure (CI) to process these data. Nevertheless, there remains a disconnect between large facilities and CI that requires researchers to be actively part of the data processing cycle. The increasing complexity of CI and data scale necessitates new data delivery models, those that can autonomously integrate large‐scale scientific facilities and CI to deliver real‐time data and insights. In this paper, we present our initial efforts using the Ocean Observatories Initiative project as a use case. In particular, we present a subscription‐based data streaming service for data delivery that leverages the Apache Kafka data streaming platform. We also show how our solution can automatically integrate large‐scale facilities with CI services for automated data processing.