Quantifying chatbots' ability to learn business processes. Issue 113 (January 2023)

• Record Type: Journal Article
• Authors: Kecht, Christoph; Egger, Andreas; Kratsch, Wolfgang; Röglinger, Maximilian
• Is Part Of: Information systems. Issue 113(2023)
• Publication Date: 2023-01
• Subjects: Chatbots -- Process mining -- Natural language processing -- Conformance checking; Database management -- Periodicals; Electronic data processing -- Periodicals; Bases de données -- Gestion -- Périodiques; Informatique -- Périodiques; Database management; Electronic data processing; Periodicals; 005.7
• DOI: 10.1016/j.is.2023.102176
• Languages: English
• ISSNs: 0306-4379

Abstract

Abstract: Chatbots enable organizations in the business-to-customer domain to respond to repetitive requests efficiently. Extant approaches in Natural Language Processing (NLP) already address the essential requirement of understanding user input and synthesizing a response as close as possible to a response a human interlocutor would give. However, we argue that the organizational adoption of chatbots further depends on the underlying model's capability to learn and comply with organizations' business processes, for example, authenticating a customer before providing sensitive details. To address this issue, we develop an approach that quantifies chatbots' ability to learn business processes using standardized process mining metrics. We demonstrate our approach by training chatbots on a dataset of more than 500, 000 customer service conversations from three companies on Twitter and show how our approach supports the quantification of a chatbot's overall ability to learn business processes from the training data. Furthermore, we quantify a chatbot's ability to learn a particular variant of the underlying process and we show how to compare the chatbot's executed steps against a given normative process model. Our approach that seamlessly integrates with existing approaches to evaluate NLP-based chatbots mitigates the current hurdles that practitioners face and, therefore, strives to foster the adoption of chatbots in practice. Highlights: Evaluates chatbots' ability to learn business processes from training data. Applies conformance checking, a particular use case of process mining. Develops and evaluates the method using the Design Science Research methodology. Uses a dataset of Twitter conversations of three companies for evaluation. Bridges the gap between Natural Language Processing research and process science.