Solving the scalarization issues of Advantage-based Reinforcement Learning algorithms. (June 2021)

• Record Type: Journal Article
• Authors: Galatolo, Federico A.; Cimino, Mario G.C.A.; Vaglini, Gigliola
• Is Part Of: Computers & electrical engineering. Volume 92(2021)
• Publication Date: 2021-06
• Subjects: Reinforcement learning -- Actor–critic -- Deep learning -- Gradient-based optimization; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2021.107117
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: In this research, some of the issues that arise from the scalarization of the multi-objective optimization problem in the Advantage Actor–Critic (A2C) reinforcement learning algorithm are investigated. The paper shows how a naive scalarization can lead to gradients overlapping. Furthermore, the possibility that the entropy regularization term can be a source of uncontrolled noise is discussed. With respect to the above issues, a technique to avoid gradient overlapping is proposed, while keeping the same loss formulation. Moreover, a method to avoid the uncontrolled noise, by sampling the actions from distributions with a desired minimum entropy, is investigated. Pilot experiments have been carried out to show how the proposed method speeds up the training. The proposed approach can be applied to any Advantage-based Reinforcement Learning algorithm. Graphical abstract: Highlights: Issues of the Advantage Actor–Critic (A2C) gradient-based optimization are addressed. The proposed variant of the A2C avoids gradient overlapping and controls the entropy. The speedup with respect to A2C is better for increasing environment complexity.