Learning offline: memory replay in biological and artificial reinforcement learning. Issue 10 (October 2021)

• Record Type: Journal Article
• Authors: Roscow, Emma L.; Chua, Raymond; Costa, Rui Ponte; Jones, Matt W.; Lepora, Nathan
• Is Part Of: Trends in neurosciences. Volume 44:Issue 10(2021)
• Page Start: 808
• Page End: 821
• Publication Date: 2021-10
• Subjects: computation -- deep neural networks -- hippocampus -- Q-learning -- reward; Neurology -- Periodicals; Neurophysiology -- Periodicals; Neurobiology -- Periodicals; 612.8
• DOI: 10.1016/j.tins.2021.07.007
• Languages: English
• ISSNs: 0166-2236

Abstract

Abstract : Learning to act in an environment to maximise rewards is among the brain's key functions. This process has often been conceptualised within the framework of reinforcement learning, which has also gained prominence in machine learning and artificial intelligence (AI) as a way to optimise decision making. A common aspect of both biological and machine reinforcement learning is the reactivation of previously experienced episodes, referred to as replay. Replay is important for memory consolidation in biological neural networks and is key to stabilising learning in deep neural networks. Here, we review recent developments concerning the functional roles of replay in the fields of neuroscience and AI. Complementary progress suggests how replay might support learning processes, including generalisation and continual learning, affording opportunities to transfer knowledge across the two fields to advance the understanding of biological and artificial learning and memory. Highlights: Reinforcement learning in deep neural networks often relies on the interleaving of new and old episodes, a technique which mimics the replay of neuronal activity in the brain. Biological replay is important for memory consolidation and has wider roles in other cognitive processes such as planning and generalisation. Deep neural networks offer a framework for understanding the role of replay in learning and memory. We suggest how recent developments could be leveraged to support more robust, efficient, and flexible reinforcement learning agents by avoiding explicit storage of past trials. Theoretical advances and more sophisticated experimental task designs will help uncover how biological replay supports complex cognition over time and throughout the brain.