Stratified rank histograms for ensemble forecast verification under serial dependence. (24th April 2020)

• Record Type: Journal Article
• Authors: Bröcker, Jochen; Ben Bouallègue, Zied
• Is Part Of: Quarterly journal of the Royal Meteorological Society. Volume 146:Number 729(2020)
• Page Start: 1976
• Page End: 1990
• Publication Date: 2020-04-24
• Subjects: ensemble forecasts -- forecast evaluation -- rank histograms -- reliability -- serial dependence -- statistical methods; Meteorology -- Periodicals; 551.5
• DOI: 10.1002/qj.3778
• Languages: English
• ISSNs: 0035-9009

Abstract

Abstract: Rank histograms are a popular way to assess the reliability of ensemble forecasting systems. If the ensemble forecasting system is reliable, the rank histogram should be flat, "up to statistical fluctuations." There are two long‐noted challenges to this approach. Firstly, uniformity of the overall distribution is implied by but does not imply reliability; ideally the distribution of the ranks should be uniform even conditionally on different forecast scenarios. Secondly, the ranks are serially dependent in general, precluding the use of standard goodness‐of‐fit tests to assess the uniformity of rank distributions without any further precautions. The present paper deals with both these issues by drawing together the concept of stratified rank histograms, which have been developed to deal with the first issue, with ideas that exploit the reliability condition to manage the serial correlations, thus dealing with the second issue. As a result, tests for uniformity of stratified rank histograms are presented that are valid under serial correlations. Abstract : The figure shows rank histograms for an ensemble forecasting system for 2 m temperature at 1200 UTC measured at Beauvais (France). The three histograms all refer to the same forecasting system and verification but correspond to different values of the joint mean of ensemble members and verification: cold, medium, and warm conditions. The histograms suggests a conditional forecast bias, and a statistical methodology is presented whereby the reliability of ensemble forecasts based on stratified rank histograms can be tested. In particular, the methodology takes into account that the ranks are not necessarily temporally independent even for perfectly reliable forecasting systems.