In‐place query driven big data platform: Applications to post processing of environmental monitoring. (29th March 2017)

• Record Type: Journal Article
• Authors: Lee, Jen‐Gaw; Tsai, Whey‐Fone; Lee, Lung‐Cheng; Lin, Ching‐Yao; Lin, Hsi‐Ching; Tsuang, Ben‐Jei
• Other Names: Plale Beth guestEditor.; Chen Maio guestEditor.; Escudero‐Sahuquillo Jesús guestEditor.; Garcia Pedro Javier guestEditor.
• Is Part Of: Concurrency and computation. Volume 29:Number 13(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2017-03-29
• Subjects: air pollution -- big data platform -- climate model -- distributed database -- in‐place computing -- parallel visualization; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.4135
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: This paper describes the use of an experimental big data platform for applications of environmental monitoring associated with visualization of global climate forecast data and air quality model simulation and response. Environmental monitoring in general requires both capabilities of model simulation for forecast, and data processing for visualization and analyses. The in‐place query driven big data platform, based on concepts of Query Driven Visualization and shared‐nothing distributed database, thus is developed for the need. The system architecture of this experimental big data platform entails one master data node and 17 slave data nodes, while the system links to the National Center for High ‐ performance Computing supercomputer, Advanced Large‐scale Parallel Supercluster, and storage pool. For software implementation, the openSUSE operating system and MariaDB database are installed on all nodes. The master data node is responsible for metadata management and information integration and the 17 slave data nodes for distributed database and parallel model simulation, data visualization, and analyses. The application of global climate data visualization (Outgoing Longwave Radiation or OLR, temperature, rainfall, etc.) in the platform serves first to partition Network Common Data Form file data into shared‐nothing distributed databases for partial visualization in slave data nodes, then integrated into whole visualization in the master node through Message Passing Interface communication. For the application of air quality management, we first accessed Taiwan Environmental Protection Administration (EPA) observed data in the master node. EPA observed data are replicated to distributed databases in slave nodes; and the air pollution model, Gaussian plume trajectory model, is replicated in all slave nodes for model simulation, which produces output data and associated image files in the local file system. The master node is able to collect whole image files through the remote shared file system for display of the results. We can see the approach of data I/O access in 2 applications, due to individual problem features, each application is unique. Examples of benchmark cases reveal strong performance in accelerating computing speed and reducing the I/O operational time. It is found that the platform is able to accelerate climate data visualization processes, help research scientists gain the deep insights into data, and explore the potential phenomena and features, such as formation of Typhoon eddies. In air quality management applications, the platform is used to perform the air pollution model Gaussian plume trajectory model. Backward trajectory simulation of PM2.5 concentrations is used to identify the 30+ point source's contribution on 73 EPA monitor stations (receptors) in Taiwan. A user‐friendly, web‐service based big data presentation uses the heterogeneous observed and forecast pollutant data in space and time. The results support for air quality decision‐making and emergency response. The limitation of data size for applications in the platform, the current users and future development of the platform, and the linkage of PRAGMA collaboration are also described in the paper.