Designing a virtual patient dialogue system based on terminology-rich resources: Challenges and evaluation. (15th March 2020)

• Record Type: Journal Article
• Authors: Campillos-Llanos, Leonardo; Thomas, Catherine; Bilinski, Éric; Zweigenbaum, Pierre; Rosset, Sophie
• Is Part Of: Natural language engineering. Volume 26:Part 2(2020)
• Page Start: 183
• Page End: 220
• Publication Date: 2020-03-15
• Subjects: Dialogue system, -- Virtual patient, -- Terminology, -- Language resources; Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324919000329
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: Virtual patient software allows health professionals to practise their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history-taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task. We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we designed a patient record model, a knowledge model for the task and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowledge. We used a frame- and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language. We adopted a comprehensive approach to collect terms and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161, 000 terms, and dictionaries with over 959, 000 word/concept entries. We assessed our approach by having 71 participants (39 medical doctors and 32 non-medical evaluators) interact with the system and use 35 cases from 18 specialities. We conducted a quantitative evaluation of all components by analysing interaction logs (11, 834 turns). Natural language understanding achieved an F-measure of 95.8%. Dialogue management provided on average 74.3 (±9.5)% of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All evaluated aspects obtained mean scores above the Likert mid-scale point. We analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8% of their terms. Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.