Improved Hadoop-based cloud for complex model simulation optimization: Calibration of SWAT as an example. (March 2022)

• Record Type: Journal Article
• Authors: Ma, Jinfeng; Rao, Kaifeng; Li, Ruonan; Yang, Yanzheng; Li, Weifeng; Zheng, Hua
• Is Part Of: Environmental modelling & software. Volume 149(2022)
• Publication Date: 2022-03
• Subjects: Hadoop-based cloud -- Sequential model -- Parallel computing -- Partial failure -- Simulation optimization -- SWAT; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2022.105330
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: A simulation optimization framework requires a substantial number of model simulations, which are computationally intensive and may be impractical when the model simulations are extremely time-consuming. This paper presents an improved Hadoop-based cloud framework to alleviate the computational burden of optimization. The framework parallelizes conventional sequential-model-based optimization techniques by concurrently orchestrating multiple model computations within Hadoop MapReduce. It guarantees the reliability of simulation optimization tasks by handling node failures without affecting the ongoing simulation. A case study, using Bayesian optimization to calibrate a SWAT model, achieved a speedup of nearly 55–58 when using 100 cores, demonstrating the efficiency of parallelizing the Bayesian optimization algorithm on the Hadoop-based cloud. Experiments in which computing nodes were dynamically increased or decreased demonstrated that the framework can automatically rebalance the workload across the remaining nodes. The framework is readily adaptable to other complex model applications that perform sequential-model-based optimizations or large-scale simulations. Highlights: Hadoop-based framework to parallelize sequential-model-based optimization methods was proposed. Coupling between complex model and MapReduce framework without reducer procedure was simplified. The framework elegantly handles partial failure during simulation optimization. The ability to reduce execution time and handle partial failure was demonstrated. The framework is readily adaptable to other complex optimization applications.