An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle. (1st December 2021)

• Record Type: Journal Article
• Authors: Yang, Ningkang; Han, Lijin; Xiang, Changle; Liu, Hui; Li, Xunmin
• Is Part Of: Energy. Volume 236(2021)
• Publication Date: 2021-12-01
• Subjects: Hybrid electric vehicle -- Real-time energy management -- Indirect reinforcement learning -- High-order Markov chain; Power resources -- Periodicals; Power (Mechanics) -- Periodicals; Energy consumption -- Periodicals; 333.7905
• DOI: 10.1016/j.energy.2021.121337
• Languages: English
• ISSNs: 0360-5442

Abstract

Abstract: This paper proposes a real-time indirect reinforcement learning based strategy to reduce the fuel consumption. In order to improve the real-time performance and achieve learning online, the simulated experience from environment model is adopted for the learning process, which is called indirect reinforcement learning. To establish an accurate environment model, a high-order Markov Chain is introduced and detailed, which is more precise than a widely used first-order Markov Chain. Corresponding with the model, how the reinforcement learning algorithm learns from the simulated experience is illustrated. Furthermore, an online recursive form of the transition probability matrix is derived, through which the statistical characteristics from the practical driving conditions can be collected. The induced matrix norm is chosen as a criterion to quantify the differences between the transition probability matrices and to determine the time for updating the environment model and triggering the recalculation of the reinforcement learning algorithm. Simulation results demonstrate that, compared with the direct RL, the proposed strategy can effectively reduce the learning time while maintains satisfied fuel economy. Furthermore, a hardware-in-the-loop experiment verifies its real-time capability and actual applicability. Highlights: A parsimonious high order Markov Chain model is introduced and utilized. An indirect reinforcement learning algorithm is developed. The power transition probability of the Markov Chain is updated online and recursively. The introduced matrix norm is used to determine the update of the control policy. The proposed strategy improves fuel economy and operates in real time.