Estimate the limit of predictability in short-term traffic forecasting: An entropy-based approach. (May 2022)

• Record Type: Journal Article
• Authors: Li, Guopeng; Knoop, Victor L.; van Lint, Hans
• Is Part Of: Transportation research. Volume 138(2022)
• Publication Date: 2022-05
• Subjects: Traffic forecasting -- Information theory -- Conditional differential entropy -- Predictability analysis; Transportation -- Periodicals; Transportation -- Technological innovations -- Periodicals; 388.011
• DOI: 10.1016/j.trc.2022.103607
• Languages: English
• ISSNs: 0968-090X

Abstract

Abstract: Accurate short-term traffic forecasting is the cornerstone for Intelligent Transportation Systems. In the past several decades, many models have been proposed to continuously improve the predictive accuracy. A key but unsolved question is whether there is a theoretical bound to the accuracy with which traffic can be predicted and whether that limit can be directly estimated from data. To answer this question, we use core concepts in information theory to derive the limit of predictability in short-term traffic forecasting. Theoretical analysis proves that conditional differential entropy poses a rigorous lower bound of negative-log-likelihood (NLL) for probabilistic models. And the continuous form of Fano's theorem further gives a loose lower bound of mean-square-error (MSE) for deterministic models. Based on the special properties of traffic dynamics, two assumptions are made in the estimate of entropy metrics: cyclostationarity (traffic phenomena show strong periodicity) and localized spatial correlation (due to kinematic wave propagation). They allow formulating the limit of predictability as a function of longitudinal space and time-of-day which finds the most uncertain locations and periods solely from data. Experiments on univariate traffic accumulation forecasting and network-level speed forecasting show that the selected models, including some state-of-the-art deep learning models, indeed cannot outperform the estimated lower bounds but just approach them. The limit of predictability depends on time-of-day, network locations, observation range, and prediction horizon. The results reveal that the stochastic nature of traffic dynamics and improper assumptions on the prior distribution of output are two major factors that restrict the predictive performance. In summary, the proposed method estimates a trustworthy performance boundary for most traffic forecasting models. These conclusions are helpful for further studies in this domain. Highlights: Estimate the limit of predictability for data-driven traffic forecasting. Embed spatiotemporal correlations in the lower bound estimation scheme. Identify the most unpredictable time slots and locations directly from data. Clarify the major bottlenecks for further improving predictive accuracy.