Solving the scalarization issues of Advantage-based Reinforcement Learning algorithms. (June 2021) - Galatolo, Federico A.; Cimino, Mario G.C.A.; Vaglini, Gigliola
Abstract: In this research, some of the issues that arise from the scalarization of the multi-objective optimization problem in the Advantage Actor–Critic (A2C) reinforcement learning algorithm are investigated. The paper shows how a naive scalarization can lead to gradients overlapping. Furthermore, the possibility that the entropy regularization term can be a source of uncontrolled noise is discussed. With respect to the above issues, a technique to avoid gradient overlapping is proposed, while keeping the same loss formulation. Moreover, a method to avoid the uncontrolled noise, by sampling the actions from distributions with a desired minimum entropy, is investigated. Pilot experiments have been carried out to show how the proposed method speeds up the training. The proposed approach can be applied to any Advantage-based Reinforcement Learning algorithm. Graphical abstract: Highlights: Issues of the Advantage Actor–Critic (A2C) gradient-based optimization are addressed. The proposed variant of the A2C avoids gradient overlapping and controls the entropy. The speedup with respect to A2C is better for increasing environment complexity.