A Comprehensive Python Toolkit for Accessing High‐Throughput Computing to Support Large Hydrologic Modeling Tasks. (13th September 2016)

• Record Type: Journal Article
• Authors: Christensen, Scott D.; Swain, Nathan R.; Jones, Norman L.; Nelson, E. James; Snow, Alan D.; Dolder, Herman G.
• Is Part Of: Journal of the American Water Resources Association. Volume 53:Number 2(2017:Apr.)
• Page Start: 333
• Page End: 343
• Publication Date: 2016-09-13
• Subjects: high‐throughput computing -- computational methods -- decision support systems -- simulation -- Python -- cloud computing; Water-supply -- Periodicals; Hydrology -- Periodicals; Water resources development -- Periodicals; Water resources development -- Environmental aspects -- Periodicals; 333.9100973
• DOI: 10.1111/1752-1688.12455
• Languages: English
• ISSNs: 1093-474X

Abstract

Abstract: The National Flood Interoperability Experiment (NFIE) was an undertaking that initiated a transformation in national hydrologic forecasting by providing streamflow forecasts at high spatial resolution over the whole country. This type of large‐scale, high‐resolution hydrologic modeling requires flexible and scalable tools to handle the resulting computational loads. While high‐throughput computing (HTC) and cloud computing provide an ideal resource for large‐scale modeling because they are cost‐effective and highly scalable, nevertheless, using these tools requires specialized training that is not always common for hydrologists and engineers. In an effort to facilitate the use of HTC resources the National Science Foundation (NSF) funded project, CI‐WATER, has developed a set of Python tools that can automate the tasks of provisioning and configuring an HTC environment in the cloud, and creating and submitting jobs to that environment. These tools are packaged into two Python libraries: CondorPy and TethysCluster. Together these libraries provide a comprehensive toolkit for accessing HTC to support hydrologic modeling. Two use cases are described to demonstrate the use of the toolkit, including a web app that was used to support the NFIE national‐scale modeling.