A deep deterministic policy gradient algorithm based on averaged state-action estimation. (July 2022)

• Record Type: Journal Article
• Authors: Xu, Jian; Zhang, Haifei; Qiu, Jianlin
• Is Part Of: Computers & electrical engineering. Volume 101(2022)
• Publication Date: 2022-07
• Subjects: Deep reinforcement learning -- Deep deterministic policy gradients -- Averaged state-action estimation -- Target approximate error; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2022.108015
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Deep Reinforcement Learning (DRL), one of the most popular research topics in artificial intelligence, has achieved a breakthrough in continuous control tasks. Nonetheless, the DRL algorithm's instability and local optimality have a bad influence impact on its performance. The Deep Deterministic Policy Gradients (DDPG) algorithm uses a "soft" update to slow down the target value rate of change to alleviate this problem. However, there is still a specific target approximate error variance. The variance will aggravate the degree of the data dispersion and reduce the stability of the model. This paper proposed the DDPG with averaged state-action estimation (Averaged-DDPG) algorithm. It aims to minimize the adverse effects of conflict, which calculates the action reward by averaging the estimated values of previously learned Q values, thus reducing the training process's fluctuation and improving the algorithm's performance. The evaluation results in continuous control tasks show that Averaged-DDPG can enhance the agent's learning efficiency and training balance more effectively than the original DDPG algorithm.