Whittle index based Q-learning for restless bandits with average reward. (May 2022)

• Record Type: Journal Article
• Authors: Avrachenkov, Konstantin E.; Borkar, Vivek S.
• Is Part Of: Automatica. Volume 139(2022)
• Publication Date: 2022-05
• Subjects: Discrete event system -- Reinforcement learning -- Restless bandits -- Whittle index -- Q-learning -- Average reward; Automatic control -- Periodicals; Automation -- Periodicals; 629.805
• DOI: 10.1016/j.automatica.2022.110186
• Languages: English
• ISSNs: 0005-1098

Abstract

Abstract: A novel reinforcement learning algorithm is introduced for multiarmed restless bandits with average reward, using the paradigms of Q-learning and Whittle index. Specifically, we leverage the structure of the Whittle index policy to reduce the search space of Q-learning, resulting in major computational gains. Rigorous convergence analysis is provided, supported by numerical experiments. The numerical experiments show excellent empirical performance of the proposed scheme.