Distributed Gradient Temporal Difference Off-policy Learning With Eligibility Traces: Weak Convergence. Issue 2 (2020)

• Record Type: Journal Article
• Authors: Stanković, Miloš S.; Beko, Marko; Stanković, Srdjan S.
• Is Part Of: IFAC-PapersOnLine. Volume 53:Issue 2(2020)
• Page Start: 1563
• Page End: 1568
• Publication Date: 2020
• Subjects: Reinforcement learning -- Distributed consensus -- Value function approximation -- Convergence -- Eligibility traces -- Off-policy learning -- Weak convergence -- Multi-agent systems; Automatic control -- Periodicals; 629.805
• DOI: 10.1016/j.ifacol.2020.12.2184
• Languages: English
• ISSNs: 2405-8963

Abstract

Abstract: In this paper we propose two novel distributed algorithms for multi-agent off-policy learning of linear approximation of the value function in Markov decision processes. The algorithms differ in the way of how distributed consensus iterations are incorporated in a basic, recently proposed, single agent scheme. The proposed completely decentralized off-policy learning schemes subsume local eligibility traces, and allow applications in which all the agents may have different behavior policies while evaluating a single target policy. Under nonrestrictive assumptions on the time-varying network topology and the individual state-visiting distributions of the agents, we prove that the parameter estimates of the algorithms weakly converge to a consensus. The variance reduction properties of the proposed algorithms are demonstrated. We also formulate specific guidelines on how to design the network weights and topology. The results are illustrated using simulations.