FritzBot: A data-driven conversational agent for physical-computing system design. Issue 155 (November 2021)

• Record Type: Journal Article
• Authors: Chen, Taizhou; Xu, Lantian; Zhu, Kening
• Is Part Of: International journal of human-computer studies. Issue 155(2021)
• Publication Date: 2021-11
• Subjects: Natural-language interaction -- User interface -- Physical computing -- Design -- BiLSTM -- CRF; Human-machine systems -- Periodicals; Systems engineering -- Periodicals; Human engineering -- Periodicals; Human engineering; Human-machine systems; Systems engineering; Periodicals; Electronic journals; 004.019
• DOI: 10.1016/j.ijhcs.2021.102699
• Languages: English
• ISSNs: 1071-5819

Abstract

Highlights: We construct a lexical circuit-event database for modeling novice users' natural-language behaviors on physical-computing system design and development. For proof of concept, we develop FrtizBot, a BiLSTM-CRF-based system leveraging natural-language interaction for designing and creating Arduino based physical-computing systems. We conduct a user study showing the effectiveness of FritzBot on sup1 porting novice users' physical-computing task through natural-language interaction. Abstract: Creating physical-computing systems, especially selecting correct electronic components, assembling the circuit, and implementing the program, can be challenging for novice users. In this paper, we present FritzBot, a data-driven conversational agent supporting novice users on creating physical-computing systems through natural-language interaction. FritzBot is built upon the structure of a BiLSTM-CRF (bi-directional Long Short-term Memory Network and Conditional Random Field) neural network, as a plug-in for Fritzing. The neural network is trained on a lexical circuit-event database derived from 152 students' reports on their physical-computing course projects. By processing the user's textual description on his/her physical-computing idea, FritzBot can extract the causal relationships between the input and the output events, identify the corresponding electronic components, and generate the Arduino-based circuit and the code along with the step-by-step construction guidelines. Our user study shows that compared to the original Arduino software and the circuit-autocompletion software available in the commercial market, FritzBot significantly shortens the time spent, reduces the perceived workload, and enhances the satisfaction/joy for inexperienced users on designing and prototyping physical-computing systems.