Reinforcement-learning based dialogue system for human–robot interactions with socially-inspired rewards. (November 2015)

• Record Type: Journal Article
• Authors: Ferreira, Emmanuel; Lefèvre, Fabrice
• Is Part Of: Computer speech & language. Volume 34(2015)
• Page Start: 256
• Page End: 274
• Publication Date: 2015-11
• Subjects: Human–robot interaction -- POMDP-based dialogue management -- Reinforcement learning -- Reward shaping; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2015.03.007
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract : Highlights: We integrate user appraisals in a POMDP-based dialogue manager procedure. We employ additional socially-inspired rewards in a RL setup to guide the learning. A unified framework for speeding up the policy optimisation and user adaptation. We consider a potential-based reward shaping with a sample efficient RL algorithm. Evaluated using both user simulator (information retrieval) and user trials (HRI). Abstract: This paper investigates some conditions under which polarized user appraisals gathered throughout the course of a vocal interaction between a machine and a human can be integrated in a reinforcement learning-based dialogue manager. More specifically, we discuss how this information can be cast into socially-inspired rewards for speeding up the policy optimisation for both efficient task completion and user adaptation in an online learning setting. For this purpose a potential-based reward shaping method is combined with a sample efficient reinforcement learning algorithm to offer a principled framework to cope with these potentially noisy interim rewards. The proposed scheme will greatly facilitate the system's development by allowing the designer to teach his system through explicit positive/negative feedbacks given as hints about task progress, in the early stage of training. At a later stage, the approach will be used as a way to ease the adaptation of the dialogue policy to specific user profiles. Experiments carried out using a state-of-the-art goal-oriented dialogue management framework, the Hidden Information State (HIS), support our claims in two configurations: firstly, with a user simulator in the tourist information domain (and thus simulated appraisals), and secondly, in the context of man–robot dialogue with real user trials.