Reinforcement learning in deregulated energy market: A comprehensive review. (1st January 2023)

• Record Type: Journal Article
• Authors: Zhu, Ziqing; Hu, Ze; Chan, Ka Wing; Bu, Siqi; Zhou, Bin; Xia, Shiwei
• Is Part Of: Applied energy. Volume 329(2023)
• Publication Date: 2023-01-01
• Subjects: Energy market -- Reinforcement learning -- Bidding strategy -- Optimal dispatching; AC Actor-Critic algorithm -- ASM Ancillary Service Market -- CEAM Carbon Emission Auction Market -- CNN Convolutional Neural Network -- DA Day-Ahead -- DDPG Deep Deterministic Policy Gradient -- DP Dynamic Programming -- DQN Deep Q-learning -- DSO Distribution System Operator -- EC Edge Computing -- FL Federated Learning -- GENCO Generation Company -- HRL Hierarchical Reinforcement Learning -- ISO Independent System Operator -- LSTM Long Short Term Memory -- MADDPG Multi-Agent Deep Deterministic Policy Gradient -- MARL Multi-Agent Reinforcement Learning -- MC Monte-Carlo method -- MDP Markov Decision Process -- MG Microgrid -- NEP Nash Equilibrium Point -- P2P Peer-to-Peer -- RARL Risk-Averse Reinforcement Learning -- RDG Renewable Distributed Generators -- RL Reinforcement Learning -- RT Real-Time -- SPG Stochastic Policy Gradient -- TD Temporal Difference method -- VPP Virtual Power Plant; Power (Mechanics) -- Periodicals; Energy conservation -- Periodicals; Energy conversion -- Periodicals; 621.042
• DOI: 10.1016/j.apenergy.2022.120212
• Languages: English
• ISSNs: 0306-2619

Abstract

Highlights: Applications of Reinforcement Learning (RL) in energy market operation are reviewed. Fundamentals of RL including basic concepts and algorithms are summarized. Critical discussions on applicability and obstacles of RL deployment are provided. Advanced RL techniques developed recently are recommended as future perspectives. Abstract: The increasing penetration of renewable generations, along with the deregulation and marketization of power industry, promotes the transformation of energy market operation paradigms. The optimal bidding strategy and dispatching methodologies under these new paradigms are prioritized concerns for both market participants and power system operators. In contrast with conventional solution methodologies, the Reinforcement Learning (RL), as an emerging machine learning technique that exhibits a more favorable computational performance, is playing an increasingly significant role in both academia and industry. This paper presents a comprehensive review of RL applications in deregulated energy market operation including bidding and dispatching strategy optimization, based on more than 150 carefully selected papers. For each application, apart from a paradigmatic summary of generalized methodology, in-depth discussions of applicability and obstacles while deploying RL techniques are also provided. Finally, some RL techniques that have great potentiality to be deployed in bidding and dispatching problems are recommended and discussed.