Indices for the evaluation of wildfire spread simulations using contemporaneous predictions and observations of burnt area. (September 2016)

• Record Type: Journal Article
• Authors: Duff, Thomas J.; Chong, Derek M.; Tolhurst, Kevin G.
• Is Part Of: Environmental modelling & software. Volume 83(2016:Sep.)
• Page Start: 276
• Page End: 285
• Publication Date: 2016-09
• Subjects: Area Difference Index -- Bushfire -- Fire spread model -- PHOENIX RapidFire -- Jaccard's coefficient -- Kappa -- Precision -- Recall -- Shape Deviation Index -- Sørensen similarity -- Spatial validation -- Verification -- Wildfire; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2016.05.005
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: Methods to objectively evaluate performance are critical for model development. In contrast to recent advances in wildfire simulation, there has been limited attention to evaluating fire model performance. Information to validate fire models is typically limited, commonly to a few perimeter observations at a small number of points in time. We review metrics for comparing two burnt areas at a point in time: observed and predicted. These are compared in an idealised landscape and with a case study evaluating the performance of simulations of an Australian wildfire. We assessed: Shape Deviation Index (SDI), Jaccard's coefficient, F1, Sørensen's Similarity and Area Difference Index (ADI). For decomposing fit into error components (overprediction and underprediction) we assessed the partial indices of SDI and ADI, Precision and Recall. The various metrics were evaluated for their ability to represent error and their suitability for use in model improvement frameworks. Highlights: Fire simulation models are of increasing importance to wildfire management. Verification of fire simulation models is necessary before operational use. There are no indices of model performance being used consistently. The properties of various fire model performance indices were evaluated. Performance indices properties should be considered when interpreting results.