Agreement between visual inspection and objective analysis methods: A replication and extension. Issue 3 (27th April 2022)

• Record Type: Journal Article
• Authors: Taylor, Tessa; Lanovaz, Marc J.
• Is Part Of: Journal of applied behavior analysis. Volume 55:Issue 3(2022)
• Page Start: 986
• Page End: 996
• Publication Date: 2022-04-27
• Subjects: artificial intelligence -- conservative dual criteria -- interrater agreement -- machine learning -- visual inspection; Psychology, Applied -- Periodicals; Social Behavior -- Periodicals; 150
• DOI: 10.1002/jaba.921
• Languages: English
• ISSNs: 0021-8855

Abstract

Abstract : Behavior analysts typically rely on visual inspection of single‐case experimental designs to make treatment decisions. However, visual inspection is subjective, which has led to the development of supplemental objective methods such as the conservative dual‐criteria method. To replicate and extend a study conducted by Wolfe et al. (2018) on the topic, we examined agreement between the visual inspection of five raters, the conservative dual‐criteria method, and a machine‐learning algorithm (i.e., the support vector classifier) on 198 AB graphs extracted from clinical data. The results indicated that average agreement between the 3 methods was generally consistent. Mean interrater agreement was 84%, whereas raters agreed with the conservative dual‐criteria method and the support vector classifier on 84% and 85% of graphs, respectively. Our results indicate that both objective methods produce results consistent with visual inspection, which may support their future use.