Advancing distributed data management for the HydroShare hydrologic information system. (April 2018)

• Record Type: Journal Article
• Authors: Yi, Hong; Idaszak, Ray; Stealey, Michael; Calloway, Chris; Couch, Alva L.; Tarboton, David G.
• Is Part Of: Environmental modelling & software. Volume 102(2018)
• Page Start: 233
• Page End: 240
• Publication Date: 2018-04
• Subjects: Distributed data management -- Big data -- Data sharing -- Hydrologic information systems -- Collaborative environment -- iRODS; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2017.12.008
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: HydroShare (https://www.hydroshare.org ) is an online collaborative system to support the open sharing of hydrologic data, analytical tools, and computer models. Hydrologic data and models are often large, extending to multi-gigabyte or terabyte scale, and as a result, the scalability of centralized data management poses challenges for a system such as HydroShare. A distributed data management framework that enables distributed physical data storage and management in multiple locations thus becomes a necessity. We use the iRODS (Integrated Rule-Oriented Data System) data grid middleware as the distributed data storage and management back end in HydroShare. iRODS provides a unified virtual file system for distributed physical storages in multiple locations and enables data federation across geographically dispersed institutions around the world. In this paper, we describe the iRODS-based distributed data management approaches implemented in HydroShare to provide a practical demonstration of a production system for supporting big data in the environmental sciences. Highlights: Uses iRODS as a back end in HydroShare to facilitate replication to the off site data store for disaster recovery. Employs iRODS federation to enable a partner institution to add storage into the HydroShare distributed data storage system. Provides iRODS user space to enable users to upload large files using iRODS clients and to add them into HydroShare resources. Uses iRODS rules and commands for on-demand resource bagging to take data operation close to data for enhanced performance. Enables interoperability with other iRODS-based systems, and provides fast transfer of big data to and from supercomputers.