PRPL + pGTIOL: The marriage of a parallel processing library and a parallel I/O library for big raster data. (October 2017)

• Record Type: Journal Article
• Authors: Miao, Jinli; Guan, Qingfeng; Hu, Shujian
• Is Part Of: Environmental modelling & software. Volume 96(2017)
• Page Start: 347
• Page End: 360
• Publication Date: 2017-10
• Subjects: Raster -- Parallel computing -- Parallel I/O -- Programming library; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2017.06.031
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: Data I/O has become a major bottleneck of computational performance of geospatial analysis and modeling. In this study, a parallel GeoTIFF I/O library (pGTIOL) was developed. Through the storage mapping and data arrangement techniques, pGTIOL can operate on files in either strip or tile storage mode, read/write any sub-domain of data within the raster dataset. pGTIOL enables asynchronized I/O, which means a process can read/write its own sub-domains of data when necessary without synchronizing with other processes. pGTIOL was integrated into the parallel raster processing library (pRPL). Several pGTIOL-based data I/O functions and options were added to pRPL, while the existing functions of pRPL stay intact. Experiments showed that the integration of pRPL and pGTIOL achieved higher performance than the original pRPL that uses GDAL as the I/O interface. Therefore, pRPL + pGTIOL enables transparent parallelism for high-performance raster processing with the capability of true parallel I/O of massive raster datasets. Highlights: pGTIOL: A programming library for asynchronized parallel I/O of GeoTIFF files. Integration of a parallel raster processing library (pRPL) and pGTIOL. pRPL + pGTIOL yields higher performance than the original pRPL.