Low resource end-to-end spoken language understanding with capsule networks. (March 2021)

• Record Type: Journal Article
• Authors: Poncelet, Jakob; Renkens, Vincent; Van hamme, Hugo
• Is Part Of: Computer speech & language. Volume 66(2021)
• Publication Date: 2021-03
• Subjects: Spoken language understanding -- End-to-end -- Intent recognition -- Capsule networks -- Multitask learning; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2020.101142
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: Capsule Networks improve upon several baseline models for E2ESLU. Rate-coded capsules perform better than place-coded capsules. Capsules allow multitask learning like recognising speakers and word occurrence. Regularisation (towards reconstruction) makes capsule content more interpretable. Capsules can infer the spoken order of labels from the input without transcripts. Abstract: Designing a Spoken Language Understanding (SLU) system for command-and-control applications is challenging. Both Automatic Speech Recognition and Natural Language Understanding are language and application dependent to a great extent. Even with a lot of design effort, users often still have to know what to say to the system for it to do what they want. We propose to use an end-to-end SLU system that maps speech directly to semantics and that can be trained by the user through demonstrations. The user can teach the system a new command by uttering the command and subsequently demonstrating its meaning through an alternative interface. The system will learn the mapping from the spoken command to the task. The dependency on the user also allows different languages and non-standard or impaired speech as valid inputs. Teaching the system requires effort from the user, so it is crucial that the system learns quickly. In this paper we propose to use capsule networks for this task, which are believed to be data efficient. We discuss two architectures for using capsule networks. We analyse their performance and compare them with two baseline systems, one based on Non-negative Matrix Factorisation (NMF) which has been successful for this task and one encoder-decoder approach. We show that in most cases the capsule network performs better than the baseline systems. Furthermore, we demonstrate the versatility of the architecture by inferring speaker identity and the user's word choice through multitask learning.