The success of Conversational AI and the AI evaluation challenge it reveals. Issue 1 (31st March 2022)

• Record Type: Journal Article
• Authors: Beaver, Ian
• Is Part Of: AI magazine. Volume 43:Issue 1 (2022)
• Page Start: 139
• Page End: 141
• Publication Date: 2022-03-31
• Subjects: Artificial intelligence -- Periodicals; Artificial intelligence -- Computer programs -- Periodicals; System design -- Periodicals; Artificial intelligence -- Technological innovations -- Periodicals; Artificial intelligence -- Industrial applications -- Periodicals; Artificial intelligence -- Educational applications -- Periodicals; Intelligence artificielle -- Périodiques; Intelligence artificielle -- Logiciels -- Périodiques; Conception de systèmes -- Périodiques; Intelligence artificielle -- Innovations -- Périodiques; Intelligence artificielle -- Applications industrielles -- Périodiques; Intelligence artificielle -- Applications en éducation -- Périodiques; Artificial intelligence; Artificial intelligence -- Computer programs; Artificial intelligence -- Educational applications; Artificial intelligence -- Industrial applications; System design; Kunstmatige intelligentie; Electronic journals; Periodicals; Periodicals; 006.305
• DOI: 10.1002/aaai.12030
• Languages: English
• ISSNs: 0738-4602

Abstract

Abstract: Research interest in Conversational artificial intelligence (ConvAI) has experienced a massive growth over the last few years and several recent advancements have enabled systems to produce rich and varied turns in conversations similar to humans. However, this apparent creativity is also creating a real challenge in the objective evaluation of such systems as authors are becoming reliant on crowd worker opinions as the primary measurement of success and, so far, few papers are reporting all that is necessary for others to compare against in their own crowd experiments. This challenge is not unique to ConvAI, but demonstrates as AI systems mature in more "human" tasks that involve creativity and variation, evaluation strategies need to mature with them.