Alleviating the estimation bias of deep deterministic policy gradient via co-regularization. (November 2022)

• Record Type: Journal Article
• Authors: Li, Yao; Wang, YuHui; Gan, YaoZhong; Tan, XiaoYang
• Is Part Of: Pattern recognition. Volume 131(2022)
• Publication Date: 2022-11
• Subjects: Reinforcement learning -- Overestimation -- Underestimation -- Co-training -- Deterministic policy gradient; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.108872
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Our method dynamically alleviates the estimation biases based on the difference between the overestimated and underestimated learners. Theoretical analysis demonstrates that the estimation biases are reduced compared with the baselines. Our method achieves the most stable performance on the average reward compared with the baselines. Abstract: The overestimation in Deep Deterministic Policy Gradients (DDPG) caused by value approximation error may result in unstable policy training. Twin Delayed Deep Deterministic Policy Gradient (TD3) addresses the overestimation but suffers from the underestimation. In this paper, we propose a Co -Regularization based D eep D eterministic (CoD2) policy gradient method to mitigate the estimation bias. Two learners characterized by overestimated and underestimated biases are trained with Co-regularization to achieve this goal. The overestimated and underestimated values are updated conservatively in CoD2 for policy evaluation. Experimental results show that our method achieves comparable performance compared with other methods.