Solving large-scale multi-agent tasks via transfer learning with dynamic state representation. (11th April 2023)

• Record Type: Journal Article
• Authors: Dou, Lintao; Jia, Zhen; Huang, Jian
• Is Part Of: International journal of advanced robotic systems. Volume 20:Number 2(2023)
• Publication Date: 2023-04-11
• Subjects: Large-scale agents -- MARL -- QMIX -- transfer learning -- UAVs; Robotics -- Periodicals; Robotics; Periodicals; 629.892
• DOI: 10.1177/17298806231162440
• Languages: English
• ISSNs: 1729-8806

Abstract

Many research results have emerged in the past decade regarding multi-agent reinforcement learning. These include the successful application of asynchronous advantage actor-critic, double deep Q-network and other algorithms in multi-agent environments, and the more representative multi-agent training method based on the classical centralized training distributed execution algorithm QMIX. However, in a large-scale multi-agent environment, training becomes a major challenge due to the exponential growth of the state-action space. In this article, we design a training scheme from small-scale multi-agent training to large-scale multi-agent training. We use the transfer learning method to enable the training of large-scale agents to use the knowledge accumulated by training small-scale agents. We achieve policy transfer between tasks with different numbers of agents by designing a new dynamic state representation network, which uses a self-attention mechanism to capture and represent the local observations of agents. The dynamic state representation network makes it possible to expand the policy model from a few agents (4 agents, 10 agents) task to large-scale agents (16 agents, 50 agents) task. Furthermore, we conducted experiments in the famous real-time strategy game Starcraft II and the multi-agent research platform MAgent. And also set unmanned aerial vehicles trajectory planning simulations. Experimental results show that our approach not only reduces the time consumption of a large number of agent training tasks but also improves the final training performance.