Decentralized control and local information for robust and adaptive decentralized Deep Reinforcement Learning. (December 2021)

• Record Type: Journal Article
• Authors: Schilling, Malte; Melnik, Andrew; Ohl, Frank W.; Ritter, Helge J.; Hammer, Barbara
• Is Part Of: Neural networks. Volume 144(2021)
• Page Start: 699
• Page End: 725
• Publication Date: 2021-12
• Subjects: Deep Reinforcement Learning -- Motor control -- Decentralization -- Local information; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2021.09.017
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Decentralization is a central characteristic of biological motor control that allows for fast responses relying on local sensory information. In contrast, the current trend of Deep Reinforcement Learning (DRL) based approaches to motor control follows a centralized paradigm using a single, holistic controller that has to untangle the whole input information space. This motivates to ask whether decentralization as seen in biological control architectures might also be beneficial for embodied sensori-motor control systems when using DRL. To answer this question, we provide an analysis and comparison of eight control architectures for adaptive locomotion that were derived for a four-legged agent, but with their degree of decentralization varying systematically between the extremes of fully centralized and fully decentralized. Our comparison shows that learning speed is significantly enhanced in distributed architectures—while still reaching the same high performance level of centralized architectures—due to smaller search spaces and local costs providing more focused information for learning. Second, we find an increased robustness of the learning process in the decentralized cases—it is less demanding to hyperparameter selection and less prone to becoming trapped in poor local minima. Finally, when examining generalization to uneven terrains —not used during training—we find best performance for an intermediate architecture that is decentralized, but integrates only local information from both neighboring legs. Together, these findings demonstrate beneficial effects of distributing control into decentralized units and relying on local information. This appears as a promising approach towards more robust DRL and better generalization towards adaptive behavior. Graphical abstract: Highlights: Deep Reinforcement Learning in motor control is heavily affected by overfitting. In the proposed decentralized approach, control is distributed into local modules. Training of these decentralized modules progressed much faster, driven by local decomposed rewards. Decentralized controller produced more robust behavior when tested for generalization.