Advanced artificial agents intervene in the provision of reward. Issue 3 (29th August 2022)

• Record Type: Journal Article
• Authors: Cohen, Michael K.; Hutter, Marcus; Osborne, Michael A.
• Is Part Of: AI magazine. Volume 43:Issue 3 (2022)
• Page Start: 282
• Page End: 293
• Publication Date: 2022-08-29
• Subjects: Artificial intelligence -- Periodicals; Artificial intelligence -- Computer programs -- Periodicals; System design -- Periodicals; Artificial intelligence -- Technological innovations -- Periodicals; Artificial intelligence -- Industrial applications -- Periodicals; Artificial intelligence -- Educational applications -- Periodicals; Intelligence artificielle -- Périodiques; Intelligence artificielle -- Logiciels -- Périodiques; Conception de systèmes -- Périodiques; Intelligence artificielle -- Innovations -- Périodiques; Intelligence artificielle -- Applications industrielles -- Périodiques; Intelligence artificielle -- Applications en éducation -- Périodiques; Artificial intelligence; Artificial intelligence -- Computer programs; Artificial intelligence -- Educational applications; Artificial intelligence -- Industrial applications; System design; Kunstmatige intelligentie; Electronic journals; Periodicals; Periodicals; 006.305
• DOI: 10.1002/aaai.12064
• Languages: English
• ISSNs: 0738-4602

Abstract

Abstract: We analyze the expected behavior of an advanced artificial agent with a learned goal planning in an unknown environment. Given a few assumptions, we argue that it will encounter a fundamental ambiguity in the data about its goal. For example, if we provide a large reward to indicate that something about the world is satisfactory to us, it may hypothesize that what satisfied us was the sending of the reward itself; no observation can refute that. Then we argue that this ambiguity will lead it to intervene in whatever protocol we set up to provide data for the agent about its goal. We discuss an analogous failure mode of approximate solutions to assistance games. Finally, we briefly review some recent approaches that may avoid this problem.