A Dynamic Penalty Function Approach for Constraint-Handling in Reinforcement Learning. Issue 3 (2021)

• Record Type: Journal Article
• Authors: Yoo, Haeun; Zavala, Victor M.; Lee, Jay H.
• Is Part Of: IFAC-PapersOnLine. Volume 54:Issue 3(2021)
• Page Start: 487
• Page End: 491
• Publication Date: 2021
• Subjects: Reinforcement Learning -- Penalty approach -- Dynamic Penalty -- Constraints; Automatic control -- Periodicals; 629.805
• DOI: 10.1016/j.ifacol.2021.08.289
• Languages: English
• ISSNs: 2405-8963

Abstract

Abstract: Reinforcement learning (RL) is attracting attention as an effective way to solve sequential optimization problems that involve high dimensional state/action space and stochastic uncertainties. Many such problems involve constraints expressed by inequality constraints. This study focuses on using RL to solve constrained optimal control problems. Most RL application studies have dealt with inequality constraints by adding soft penalty terms for violating the constraints to the reward function. However, while training neural networks to learn the value (or Q) function, one can run into computational issues caused by the sharp change in the function value at the constraint boundary due to the large penalty imposed. This difficulty during training can lead to convergence problems and ultimately lead to poor closed-loop performance. To address this issue, this study proposes a dynamic penalty (DP) approach where the penalty factor is gradually and systematically increased during training as the iteration episodes proceed. We first examine the ability of a neural network to represent a value function when uniform, linear, or DP functions are added to prevent constraint violation. The agent trained by a Deep Q Network (DQN) algorithm with the DP function approach was compared with agents with other constant penalty functions in a simple vehicle control problem. Results show that the proposed approach can improve the neural network approximation accuracy and provide faster convergence when close to a solution.