Assessing the reliability of ensemble forecasting systems under serial dependence. (27th November 2018)

• Record Type: Journal Article
• Authors: Bröcker, Jochen
• Is Part Of: Quarterly journal of the Royal Meteorological Society. Volume 144:Number 717(2018)
• Page Start: 2666
• Page End: 2675
• Publication Date: 2018-11-27
• Subjects: ensemble forecasts -- forecast evaluation -- rank histograms -- reliability -- serial dependence -- statistical methods; Meteorology -- Periodicals; 551.5
• DOI: 10.1002/qj.3379
• Languages: English
• ISSNs: 0035-9009

Abstract

Abstract : The problem of testing the reliability of ensemble forecasting systems is revisited. A popular tool to assess the reliability of ensemble forecasting systems (for scalar verifications) is the rank histogram; this histogram is expected to be more or less flat, since, for a reliable ensemble, the ranks are uniformly distributed among their possible outcomes. Quantitative tests for flatness (e.g. Pearson's goodness‐of‐fit test) have been suggested; without exception, however, these tests assume the ranks to be a sequence of independent random variables, which is not the case in general, as can be demonstrated with simple toy examples. In this article, tests are developed that take the temporal correlations between the ranks into account. A refined analysis exploiting the reliability property shows that the ranks still exhibit strong decay of correlations. This property is key to the analysis, and the proposed tests are valid for general ensemble forecasting systems with minimal extraneous assumptions. Abstract : Typical rank histograms for an ensemble forecasting system for short resp long lead times (top and bottom panels, resp). Both forecast systems are by construction reliable, but the bottom histogram exhibits stronger variations. This is due to correlations between the ranks at larger lead times, which are not independent. A new type of Goodness‐Of‐Fit test is developed which takes these correlations into account. No further extraneous assumptions are needed except that the ranks have to form an ergodic sequence.