How certain are our uncertainty bounds? Accounting for sample variability in Monte Carlo-based uncertainty estimates. (February 2021)

• Record Type: Journal Article
• Authors: Roy, Tirthankar; Gupta, Hoshin
• Is Part Of: Environmental modelling & software. Volume 136(2021)
• Publication Date: 2021-02
• Subjects: Prediction intervals -- Sampling variability -- Uncertainty -- Precision -- Monte Carlo simulation -- Estimation; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2020.104931
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: It is common for model-based simulations to be reported using prediction interval estimates that characterize the lack of precision associated with the simulated values. When based on Monte-Carlo sampling to approximate the relevant probability density function(s), such estimates can significantly underestimate the width of the prediction intervals, unless the sample size is sufficiently large. Using theoretical arguments supported by numerical experiments, we discuss the nature and severity of this problem, and demonstrate how better estimates of prediction intervals can be achieved by adjusting the interval width to account for the size of the sample used in its construction. Our method is generally applicable regardless of the form of the underlying probability density function, and can be particularly useful when the model is expensive to run and large samples are not available. We illustrate its use via a simple example involving conceptual modeling of the rainfall-runoff response of a catchment. Highlights: Due to sampling variability, prediction interval estimates from Monte Carlo simulations can be highly uncertain. A general approach is provided for adjusting prediction interval widths to account for the sample size used in their construction. The method is applicable for any form of probability density function and can be particularly useful when the model is expensive to run.