OpenCluster: A Flexible Distributed Computing Framework for Astronomical Data Processing. (20th December 2016)

• Record Type: Journal Article
• Authors: Wei, Shoulin; Wang, Feng; Deng, Hui; Liu, Cuiyin; Dai, Wei; Liang, Bo; Mei, Ying; Shi, Congming; Liu, Yingbo; Wu, Jingping
• Is Part Of: Publications of the Astronomical Society of the Pacific. Volume 129:Number 972(2017)
• Publication Date: 2016-12-20
• Subjects: instrumentation: interferometers -- methods: data analysis -- methods: numerical -- techniques: image processing -- techniques: interferometric; Astronomy -- Periodicals; Astronomy; Periodicals; Periodicals; 520.5
• DOI: 10.1088/1538-3873/129/972/024001
• Languages: English
• ISSNs: 0004-6280

Abstract

Abstract: The volume of data generated by modern astronomical telescopes is extremely large and rapidly growing. However, current high-performance data processing architectures/frameworks are not well suited for astronomers because of their limitations and programming difficulties. In this paper, we therefore present OpenCluster, an open-source distributed computing framework to support rapidly developing high-performance processing pipelines of astronomical big data. We first detail the OpenCluster design principles and implementations and present the APIs facilitated by the framework. We then demonstrate a case in which OpenCluster is used to resolve complex data processing problems for developing a pipeline for the Mingantu Ultrawide Spectral Radioheliograph. Finally, we present our OpenCluster performance evaluation. Overall, OpenCluster provides not only high fault tolerance and simple programming interfaces, but also a flexible means of scaling up the number of interacting entities. OpenCluster thereby provides an easily integrated distributed computing framework for quickly developing a high-performance data processing system of astronomical telescopes and for significantly reducing software development expenses.