High-performance Negative Database for Massive Data Management System of The Mingantu Spectral Radioheliograph. (22nd June 2017)

• Record Type: Journal Article
• Authors: Shi, Congming; Wang, Feng; Deng, Hui; Liu, Yingbo; Liu, Cuiyin; Wei, Shoulin
• Is Part Of: Publications of the Astronomical Society of the Pacific. Volume 129:Number 978(2017)
• Publication Date: 2017-06-22
• Subjects: massive data management -- MUSER–astronomical instrument; Astronomy -- Periodicals; Astronomy; Periodicals; Periodicals; 520.5
• DOI: 10.1088/1538-3873/aa732d
• Languages: English
• ISSNs: 0004-6280

Abstract

Abstract: As a dedicated synthetic aperture radio interferometer in China, the MingantU SpEctral Radioheliograph (MUSER), initially known as the Chinese Spectral RadioHeliograph (CSRH), has entered the stage of routine observation. More than 23 million data records per day need to be effectively managed to provide high-performance data query and retrieval for scientific data reduction. In light of these massive amounts of data generated by the MUSER, in this paper, a novel data management technique called the negative database (ND) is proposed and used to implement a data management system for the MUSER. Based on the key-value database, the ND technique makes complete utilization of the complement set of observational data to derive the requisite information. Experimental results showed that the proposed ND can significantly reduce storage volume in comparison with a relational database management system (RDBMS). Even when considering the time needed to derive records that were absent, its overall performance, including querying and deriving the data of the ND, is comparable with that of a relational database management system (RDBMS). The ND technique effectively solves the problem of massive data storage for the MUSER and is a valuable reference for the massive data management required in next-generation telescopes.