A Value‐Based Model Selection Approach for Environmental Random Variables. Issue 1 (11th January 2019)

• Record Type: Journal Article
• Authors: Müller, M. F.; Thompson, S. E.
• Is Part Of: Water resources research. Volume 55:Issue 1(2019)
• Page Start: 270
• Page End: 283
• Publication Date: 2019-01-11
• Subjects: value of information -- Bayesian framework -- flow duration curve -- hydropower -- goodness of fit -- model selection; Hydrology -- Periodicals; 333.91
• DOI: 10.1029/2018WR023000
• Languages: English
• ISSNs: 0043-1397

Abstract

Abstract: Environmental decisions with substantial social and environmental implications are regularly informed by model predictions, incurring inevitable uncertainty. The selection of a set of model predictions to inform a decision is usually based on model performance, measured by goodness‐of‐fit metrics. Yet goodness‐of‐fit metrics have a questionable relationship to a model's value to end users, particularly when validation data are themselves uncertain. For example, decisions based on flow frequency models are not necessarily improved by adopting models with the best overall goodness of fit. We propose an alternative model evaluation approach based on the conditional value of sample information, first defined in 1961, which has found extensive use in sampling design optimization but which has not previously been used for model evaluation. The metric uses observations from a validation set to estimate the expected monetary costs associated with model prediction uncertainties. A model is only considered superior to alternatives if (i) its predictions reduce these costs and (ii) sufficient validation data are available to distinguish its performance from alternative models. By describing prediction uncertainties in monetary terms, the metric facilitates the communication of prediction uncertainty by end users, supporting the inclusion of uncertainty analysis in decision making. Key Points: Goodness‐of‐fit metrics do not reliably indicate the added value of model predictions to environmental decisions Sampling uncertainty on validation data affects the reliability of model evaluation results The proposed approach assigns value to prediction uncertainties by estimating the costs of ensuing suboptimal decisions