A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning. Issue 2 (2020)

• Record Type: Journal Article
• Authors: Suttle, Wesley; Yang, Zhuoran; Zhang, Kaiqing; Wang, Zhaoran; Başar, Tamer; Liu, Ji
• Is Part Of: IFAC-PapersOnLine. Volume 53:Issue 2(2020)
• Page Start: 1549
• Page End: 1554
• Publication Date: 2020
• Subjects: consensus -- reinforcement learning control -- adaptive control of multi-agent systems; Automatic control -- Periodicals; 629.805
• DOI: 10.1016/j.ifacol.2020.12.2021
• Languages: English
• ISSNs: 2405-8963

Abstract

Abstract: This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-agent version of emphatic temporal difference learning for off-policy policy evaluation, and proves convergence under linear function approximation. The paper then leverages this result, in conjunction with a novel multi-agent off-policy policy gradient theorem and recent work in both multi-agent on-policy and single-agent off-policy actor-critic methods, to develop and give convergence guarantees for a new multi-agent off-policy actor-critic algorithm. An empirical validation of these theoretical results is given.