Using an Explainable Machine Learning Approach to Characterize Earth System Model Errors: Application of SHAP Analysis to Modeling Lightning Flash Occurrence. (19th April 2022)

• Record Type: Journal Article
• Authors: Silva, Sam J.; Keller, Christoph A.; Hardin, Joseph
• Is Part Of: Journal of advances in modeling earth systems. Volume 14:Number 4(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-04-19
• Subjects: machine learning -- artificial intelligence -- lightning -- Earth System modeling -- interpretable A.I. -- XGBoost; Geological modeling -- Periodicals; Climatology -- Periodicals; Geochemical modeling -- Periodicals; 551.5011
• DOI: 10.1029/2021MS002881
• Languages: English
• ISSNs: 1942-2466

Abstract

Abstract: Computational models of the Earth System are critical tools for modern scientific inquiry. Efforts toward evaluating and improving errors in representations of physical and chemical processes in these large computational systems are commonly stymied by highly nonlinear and complex error behavior. Recent work has shown that these errors can be effectively predicted using modern Artificial Intelligence (A.I.) techniques. In this work, we go beyond these previous studies to apply an explainable A.I. technique to not only predict model errors but also move toward understanding the underlying reasons for successful error prediction. We use XGBoost classification trees and SHapley Additive exPlanations analysis to explore the errors in the prediction of lightning occurrence in the NASA Goddard Earth Observing System model, a widely used Earth System Model. This explainable error prediction system can effectively predict the model error and indicates that the errors are strongly related to convective processes and the characteristics of the land surface. Plain Language Summary: Computer models of the Earth are very important tools in the modern Earth scientist's toolkit. Understanding when and why these models are wrong is a major challenge facing the scientific community. Work published in the last few years has shown that you can actually predict when these models are wrong using artificial intelligence (A.I.). We build on that work by applying existing fancy mathematical tools to these A.I. methods to understand why these computer models are wrong. We demonstrate this approach to predictions of lightning in a model created by NASA, and find that the lightning in the model is wrong in ways that are strongly related to convection in the atmosphere and some aspects of the land surface. Key Points: Errors in simulated lightning flash occurrence are learned through a machine learning classification approach An explainable machine learning technique is used to explore the drivers of the simulation errors This error prediction system indicates that errors are strongly related to convective processes and the characteristics of the land surface