The Four Ways to Consider Measurement Noise in Bayesian Model Selection—And Which One to Choose. Issue 11 (22nd November 2021)

• Record Type: Journal Article
• Authors: Reuschen, Sebastian; Nowak, Wolfgang; Guthke, Anneli
• Is Part Of: Water resources research. Volume 57:Issue 11(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-11-22
• Subjects: Bayesian model selection -- measurement noise -- model confusion matrix -- model ranking -- hydraulic tomography -- groundwater modeling; Hydrology -- Periodicals; 333.91
• DOI: 10.1029/2021WR030391
• Languages: English
• ISSNs: 0043-1397

Abstract

Abstract: Bayesian model selection (BMS) is a statistically rigorous approach to assess the plausibility of competing models. It naturally accounts for uncertainties in models and data. In this study, we discuss the role of measurement noise in BMS deeper than in past literature. We distinguish between four cases, accounting for noise in models and/or data: (1) no‐no, (2) no‐yes, (3) yes‐no, and (4) yes‐yes. These cases differ mathematically and philosophically. Only two out of these four cases are logically consistent, and they represent two potentially conflicting research questions: "Which model is best in modeling the pure physics?" (Case 1) and "which model is best in predicting the data‐generating process (i.e., physics plus noise)?" (Case 4). If we are interested in the "pure physics question, " we face two practical challenges: First, we would need noise‐free data, which is impossible to obtain; and second, the numerical approximation of Bayesian model evidence can be hard when neglecting noise. We discuss how to address both challenges and reveal that a fallback to the easier "data‐generation question" as a proxy for the "physics question" is not appropriate. We demonstrate on synthetic scenarios and a real‐world hydrogeological case study that the choice of the case has a significant impact on the outcome of posterior model weights, and hence on results of the model ranking, model selection, model averaging, model confusion analysis, and uncertainty quantification. Reality might force us to use a different case than philosophy would suggest, and we provide guidance on how to interpret model probabilities under such conditions. Key Points: We distinguish four cases of how to consider measurement noise in models and/or data within Bayesian model selection (BMS) We demonstrate with synthetic and real‐world scenarios that the choice of the case significantly affects the outcome of Bayesian model weights Finally, we recommend a BMS procedure that ensures philosophically consistent and numerically stable results under noise