User decisions, and how these could guide developments in probabilistic forecasting. (5th August 2020)

• Record Type: Journal Article
• Authors: Rodwell, M. J.; Hammond, J.; Thornton, S.; Richardson, D. S.
• Is Part Of: Quarterly journal of the Royal Meteorological Society. Volume 146:Number 732(2020)
• Page Start: 3266
• Page End: 3284
• Publication Date: 2020-08-05
• Subjects: Bayes action -- cost/loss ratio -- ensemble forecast -- proper score -- refinement -- reliability -- User Brier Score -- user decision; Meteorology -- Periodicals; 551.5
• DOI: 10.1002/qj.3845
• Languages: English
• ISSNs: 0035-9009

Abstract

Abstract: We investigate how users combine objective probabilities with their own subjective feelings when deciding how to act on weather forecast information. Results are based on two scenarios investigated at a Live Science event held by the Royal Meteorological Society. When deciding whether to go to the beach with the possibility of warm, dry weather, we find that users attempt to identify their 'Bayes Action': the one which minimises their expected negative feeling or utility. Key factors are the 'thrill' of a nice day at the beach and the 'pain' of coping with, for example, children in wet weather, and the costs of travel. The users' threshold probabilities for deciding to go to the beach thus approximately define their distribution of cost/loss ratios. This is used to calculate a 'User Brier Score' (UBS): a measure of the overall utility to society, and which could be used to guide forecast system development. When applied to operational ensemble forecasts issued by the European Centre for Medium‐Range Weather Forecasts (ECMWF) over the period 1995–2018, the UBS tends to be higher (i.e., worse) than the Brier Score, largely because users tended not to exhibit high cost/loss ratios. When deciding whether to leave a campsite in the face of potentially dangerous gales, users try to find a balance between the 'regret' of serious injury and the 'pain' of spoiling an enjoyable holiday. Some users decide to stay even at high probabilities of serious consequences – partly due to a lack of experience. On the other hand, forecasts suffer from 'complete misses' – where probabilities of zero are accompanied by non‐negligible outcome frequencies. These dominate the overall Brier Score. The frequency of complete misses halved over the period 1995–2018: a welcome improvement for users who do wish to avoid danger at low probabilities. Abstract : Participants at a Live Science event were asked to make decisions based on weather forecast probabilities and their own subjective feelings. From this we were able to estimate the participants' distribution of cost–loss ratios and evaluate a 'User Brier Score'. Being (asymptotically) proper, this user‐relevant score would also be helpful for guiding forecast system development. With the potential for dangerous gales, some users were risk‐prone; others would benefit from continued improvement in forecast refinement at low forecast probabilities.