Automatic identification and classification of Palomar Transient Factory astrophysical objects in GLADE. (2018)

• Record Type: Journal Article
• Authors: Zhao, Weijie; Rusu, Florin; Wu, Kesheng; Nugent, Peter
• Is Part Of: International journal of computational science and engineering. Volume 16:Number 4(2018)
• Page Start: 337
• Page End: 349
• Publication Date: 2018
• Subjects: parallel databases -- multi-query processing -- scientific data analysis -- similarity join -- astronomical surveys -- transient identification; Computer science -- Mathematics -- Periodicals; Computer simulation -- Mathematical aspects -- Periodicals; Computational intelligence -- Periodicals; 004.015105
• Languages: English
• ISSNs: 1742-7185

Abstract

Palomar Transient Factory (PTF) is a comprehensive detection system for the identification and classification of transient astrophysical objects. In this paper, we make two significant contributions to the PTF pipeline. First, we present an experimental study that evaluates a novel implementation of the real-time classifier in GLADE - a parallel data processing system that combines the efficiency of a database with the extensibility of map-reduce. We show how each stage in the classifier maps optimally into GLADE tasks by taking advantage of the unique features of the system - range-based data partitioning, columnar storage, multi-query execution, and in-database support for complex aggregate computation. Second, we introduce a novel parallel similarity join algorithm for advanced transient classification. We implement this algorithm in GLADE and execute it on a massive supercomputer with more than 3, 000 threads, achieving more than three orders of magnitude improvement over the PostgreSQL solution.