A Friend-or-Foe framework for multi-agent reinforcement learning policy generation in mixing cooperative–competitive scenarios. (August 2022)

• Record Type: Journal Article
• Authors: Sun, Yu; Lai, Jun; Cao, Lei; Chen, Xiliang; Xu, Zhixiong; Lian, Zhen; Fan, Huijin
• Is Part Of: Transactions of the Institute of Measurement and Control. Volume 44:Number 12(2022)
• Page Start: 2378
• Page End: 2395
• Publication Date: 2022-08
• Subjects: Deep reinforcement learning -- multi-agent reinforcement learning -- multi-agent system game theory; Automatic control -- Periodicals; Measuring instruments -- Periodicals; Commande automatique -- Périodiques; Mesure -- Instruments -- Périodiques; 681.2
• DOI: 10.1177/01423312221077755
• Languages: English
• ISSNs: 0142-3312

Abstract

Although multi-agent deep deterministic policy gradient is a classic deep reinforcement learning algorithm in multi-agent systems. It also has critical problems such as poor training stability and low policy robustness, which significantly limit the capability and application of the algorithm. So this article proposes an improved algorithm called friend-or-foe multi-agent deep deterministic policy gradient for solving the above problems. The main innovations are as follows: (1) inspired by the concept of friend-or-foe game theory, we modified the framework of the original multi-agent deep deterministic policy gradient by using two identical training networks with agents' optimal and worst actions input, which improves the robustness of training policies, and (2) we propose an action perturbation technique based on gradient-descent to expand the selection range of actions, thereby improving training stability of our proposing algorithm. Finally, we conducted multiple sets of comparative experiments between our friend-or-foe multi-agent deep deterministic policy gradient and original one in four authoritative mixed cooperative–competitive scenarios. The results show that our improving algorithm can simultaneously improve the training stability and the robustness of agents' generating policies in different complicated environments.