Hierarchical Attention Master–Slave for heterogeneous multi-agent reinforcement learning. (May 2023)

• Record Type: Journal Article
• Authors: Wang, Jiao; Yuan, Mingrui; Li, Yun; Zhao, Zihui
• Is Part Of: Neural networks. Volume 162(2023)
• Page Start: 359
• Page End: 368
• Publication Date: 2023-05
• Subjects: Multi-agent reinforcement learning -- Communication -- Heterogeneous agents -- Self-attention -- Cooperative games; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2023.02.037
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Most multi-agent reinforcement learning (MARL) approaches optimize strategy by improving itself, while ignoring the limitations of homogeneous agents that may have single function. However, in reality, the complex tasks tend to coordinate various types of agents and leverage advantages from one another. Therefore, it is a vital research issue how to establish appropriate communication among them and optimize decision. To this end, we propose a Hierarchical Attention Master–Slave (HAMS) MARL, where the Hierarchical Attention balances the weight allocation within and among clusters, and the Master–Slave architecture endows agents independent reasoning and individual guidance. By the offered design, information fusion, especially among clusters, is implemented effectively, and excessive communication is avoided, moreover, selective composed action optimizes decision. We evaluate the HAMS on both small and large scale heterogeneous StarCraft II micromanagement tasks. The proposed algorithm achieves the exceptional performance with more than 80% win rates in all evaluation scenarios, which obtains an impressive win rate of over 90% in the largest map. The experiments demonstrate a maximum improvement in win rate of 47% over the best known algorithm. The results show that our proposal outperforms recent state-of-the-art approaches, which provides a novel idea for heterogeneous multi-agent policy optimization. Highlights: Hierarchical communication structure improves information interaction effectively. Master–Slave structure endows agents independent reasoning and personal guidance. Attention graph framework accelerates learning interactive relationship in agents. Performance of algorithm promotes prominently on large heterogeneous environments.