Cloud archiving and data mining of High-Resolution Rapid Refresh forecast model output. (December 2017)

• Record Type: Journal Article
• Authors: Blaylock, Brian K.; Horel, John D.; Liston, Samuel T.
• Is Part Of: Computers & geosciences. Volume 109(2017)
• Page Start: 43
• Page End: 50
• Publication Date: 2017-12
• Subjects: Object data storage -- Data stewardship -- Atmospheric modeling -- Cloud computing; Environmental policy -- Periodicals; 550.5
• DOI: 10.1016/j.cageo.2017.08.005
• Languages: English
• ISSNs: 0098-3004

Abstract

Abstract: Weather-related research often requires synthesizing vast amounts of data that need archival solutions that are both economical and viable during and past the lifetime of the project. Public cloud computing services (e.g., from Amazon, Microsoft, or Google) or private clouds managed by research institutions are providing object data storage systems potentially appropriate for long-term archives of such large geophysical data sets. We illustrate the use of a private cloud object store developed by the Center for High Performance Computing (CHPC) at the University of Utah. Since early 2015, we have been archiving thousands of two-dimensional gridded fields (each one containing over 1.9 million values over the contiguous United States) from the High-Resolution Rapid Refresh (HRRR) data assimilation and forecast modeling system. The archive is being used for retrospective analyses of meteorological conditions during high-impact weather events, assessing the accuracy of the HRRR forecasts, and providing initial and boundary conditions for research simulations. The archive is accessible interactively and through automated download procedures for researchers at other institutions that can be tailored by the user to extract individual two-dimensional grids from within the highly compressed files. Characteristics of the CHPC object storage system are summarized relative to network file system storage or tape storage solutions. The CHPC storage system is proving to be a scalable, reliable, extensible, affordable, and usable archive solution for our research. Graphical abstract: Highlights: High resolution weather model output is archived in an object data storage system. Object storage is an affordable, useable, and reliable long-term archive solution. High impact weather events used to illustrate efficient data retrieval from archive. Model output archive makes it possible to initialize weather research simulations.