Potential-based multiobjective reinforcement learning approaches to low-impact agents for AI safety. (April 2021)

• Record Type: Journal Article
• Authors: Vamplew, Peter; Foale, Cameron; Dazeley, Richard; Bignold, Adam
• Is Part Of: Engineering applications of artificial intelligence. Volume 100(2021)
• Publication Date: 2021-04
• Subjects: Safe reinforcement learning -- Multiobjective reinforcement learning -- AI safety -- Potential-based rewards -- Low-impact agents -- Reward engineering -- Side-effects; Engineering -- Data processing -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Ingénierie -- Informatique -- Périodiques; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; Artificial intelligence; Engineering -- Data processing; Expert systems (Computer science); Periodicals; 620.00285
• DOI: 10.1016/j.engappai.2021.104186
• Languages: English
• ISSNs: 0952-1976

Abstract

Abstract: The concept of impact-minimisation has previously been proposed as an approach to addressing the safety concerns that can arise from utility-maximising agents. An impact-minimising agent takes into account the potential impact of its actions on the state of the environment when selecting actions, so as to avoid unacceptable side-effects. This paper proposes and empirically evaluates an implementation of impact-minimisation within the framework of multiobjective reinforcement learning. The key contributions are a novel potential-based approach to specifying a measure of impact, and an examination of a variety of non-linear action-selection operators so as to achieve an acceptable trade-off between achieving the agent's primary task and minimising environmental impact. These experiments also highlight a previously unreported issue with noisy estimates for multiobjective agents using non-linear action-selection, which has broader implications for the application of multiobjective reinforcement learning.