Controlled Abstention Neural Networks for Identifying Skillful Predictions for Regression Problems. (21st December 2021)

• Record Type: Journal Article
• Authors: Barnes, Elizabeth A.; Barnes, Randal J.
• Is Part Of: Journal of advances in modeling earth systems. Volume 13:Number 12(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-12-21
• Subjects: neural networks -- uncertainty -- regression -- forecasts of opportunity -- prediction; Geological modeling -- Periodicals; Climatology -- Periodicals; Geochemical modeling -- Periodicals; 551.5011
• DOI: 10.1029/2021MS002575
• Languages: English
• ISSNs: 1942-2466

Abstract

Abstract: The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed "forecasts of opportunity." When these opportunities are not present, scientists need prediction systems that are capable of saying "I don't know." We introduce a novel loss function, termed "abstention loss, " that allows neural networks to identify forecasts of opportunity for regression problems. The abstention loss works by incorporating uncertainty in the network's prediction to identify the more confident samples and abstain (say "I don't know") on the less confident samples. The abstention loss is designed to determine the optimal abstention fraction, or abstain on a user‐defined fraction using a standard adaptive controller. Unlike many methods for attaching uncertainty to neural network predictions post‐training, the abstention loss is applied during training to preferentially learn from the more confident samples. The abstention loss is built upon nonlinear heteroscedastic regression, a standard computer science method. While nonlinear heteroscedastic regression is a simple yet powerful tool for incorporating uncertainty in regression problems, we demonstrate that the abstention loss outperforms it for the synthetic climate use cases explored here. The implementation of the proposed abstention loss is straightforward in most network architectures designed for regression, as it only requires modification of the output layer and loss function. Plain Language Summary: The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we can look for specific states of the system that lead to more predictable behavior than others, often termed "forecasts of opportunity." When these opportunities are not present, scientists need prediction systems that are capable of saying "I don't know." We present a method for teaching neural networks, a type of machine learning tool, to say "I don't know" for regression problems. By doing so, the neural network focuses less on the predictions it identifies as problematic and focuses more on the predictions where its confidence is high. In the end, this leads to better predictions. Key Points: A simple neural network approach for adding uncertainty to climate regression problems is explored A new abstention loss is introduced to identify, and preferentially learn from, more confident samples The abstention loss outperforms other regression loss approaches for multiple climate use cases