Modeling operator self-assessment in human-autonomy teaming settings. Issue 157 (January 2022)

• Record Type: Journal Article
• Authors: Cummings, Mary L.; Li, Songpo; Zhu, Haibei
• Is Part Of: International journal of human-computer studies. Issue 157(2022)
• Publication Date: 2022-01
• Subjects: Human-autonomy teaming -- UAV -- Strategy modeling -- Hidden Markov model -- Human performance -- Confidence; Human-machine systems -- Periodicals; Systems engineering -- Periodicals; Human engineering -- Periodicals; Human engineering; Human-machine systems; Systems engineering; Periodicals; Electronic journals; 004.019
• DOI: 10.1016/j.ijhcs.2021.102729
• Languages: English
• ISSNs: 1071-5819

Abstract

Highlights: Hidden Markov Models (HMMs) can be used to represent strategies employed by operators in first-person control of UAVs for inspection tasks like bridge inspections. These HMM representations captured differences in strategies for people who both succeeded and crashed, as well as those who were calibrated and overconfident in their self-assessments. People who were overconfident tended to attempt three axis control as opposed to two axis control which lead to an increased crash rate. Using HMMs to capture operator strategies could help to identify how new technologies affect operator strategy development, or how and when operators develop appropriate strategies needed in satisfactory operation of complex systems. Abstract: The need to design for appropriate human-autonomy teaming has become increasingly important as systems grow in complexity, especially those that require time-pressured interactions like in unmanned aerial vehicle (UAV) operations. However, it is not always clear whether operators develop effective strategies for computer-based technologies. When operators are given such tools, their performances can be statistically compared but often such assessments only provide summative information. These comparisons do not indicate how and why technology influenced people's strategies and actions. To fill this gap, we utilized Hidden Markov Models (HMMs) to represent strategies employed by operators in first-person control of UAVs for inspection tasks. The resulting models captured differences in strategies for people who both succeeded and crashed, as well as those who were overconfident in their self-assessments, and those who were not. People who were not overconfident exhibited less risky strategies and were more successful. These findings were further strengthened by a quantitative state similarity metric, which indicated where and for who possible interventions could improve outcomes. This application of HMMs to operator strategy representation could help to identify effective operator strategy development in the use of computer-based technologies, and what kind of interventions could be the most effective in improving outcomes.