Charging Station Management Strategy for Returns Maximization via Improved TD3 Deep Reinforcement Learning. (15th December 2022)

• Record Type: Journal Article
• Authors: Li, Hengjie; Zhu, Jianghao; Zhou, Yun; Feng, Qi; Feng, Donghan
• Other Names: Hampannavar Santoshkumar Academic Editor.
• Is Part Of: International transactions on electrical energy systems. Volume 2022(2022)
• Publication Date: 2022-12-15
• Subjects: Electric power -- Periodicals; Electric power systems -- Periodicals; Electrical engineering -- Periodicals; 621.3
• DOI: 10.1155/2022/6854620
• Languages: English
• ISSNs: 2050-7038

Abstract

Abstract : Maximizing the return on electric vehicle charging station (EVCS) operation helps to expand the EVCS, thus expanding the EV (electric vehicle) stock and better addressing climate change. However, in the face of dynamic regulation scenarios with large data, multiple variables, and low time scales, the existing regulation strategies aiming at maximizing EVCS returns many times fail to meet the demand. To handle increasingly complex regulation scenarios, a deep reinforcement learning algorithm (DRL) based on the improved twin delayed deep deterministic policy gradient (TD3) is used to construct basic energy management strategies in this paper. To enable the strategy to be more suitable for the goal of real-time energy regulation strategy, we used Thompson sampling strategy to improve TD3's exploration noise sampling strategy, which greatly accelerated the initial convergence of TD3 during training. Also, we use marginalised importance sampling to calculate the Q -return function for TD3, which ensures that the constructed strategies are more likely to learn high-value experiences while having higher robustness. It is shown in numerical experiments that the charging station management strategy (CSMS) based on the modified TD3 obtains the fastest convergence speed and the highest robustness and achieves the largest operational returns compared to the CSMS constructed using deep deterministic policy gradient (DDPG), actor-critic using Kronecker-factored trust region (ACKTR), trust region policy optimization (TRPO), proximal policy optimization (PPO), soft actor-critic (SAC), and the original TD3.