A methodology for applying students' interactive task performance scores from a multimedia-based performance assessment in a Bayesian Network. (July 2016)

• Record Type: Journal Article
• Authors: de Klerk, Sebastiaan; Eggen, Theo J.H.M.; Veldkamp, Bernard P.
• Is Part Of: Computers in human behavior. Volume 60(2016)
• Page Start: 264
• Page End: 279
• Publication Date: 2016-07
• Subjects: Multimedia-based performance assessment -- Scoring interactive student performance -- Evidence identification -- Evidence accumulation -- Log file analysis -- Bayesian network; Interactive computer systems -- Periodicals; Man-machine systems -- Periodicals; 004.019
• DOI: 10.1016/j.chb.2016.02.071
• Languages: English
• ISSNs: 0747-5632

Abstract

Abstract: Computer-based simulations are increasingly being used in educational assessment. In most cases, the simulation-based assessment (SBA) is used for formative assessment, which can be defined as assessment for learning, but as research on the topic continues to grow, possibilities for summative assessment, which can be defined as assessment of learning, are also emerging. The current study contributes to research on the latter category of assessment. In this article, we present a methodology for scoring the interactive and complex behavior of students in a specific type of SBA, namely, a Multimedia-based Performance Assessment (MBPA), which is used for a summative assessment purpose. The MBPA is used to assess the knowledge, skills, and abilities of confined space guard (CSG) students. A CSG supervises operations that are carried out in a confined space (e.g., a tank or silo). We address two specific challenges in this article: the evidence identification challenge (i.e., scoring interactive task performance), and the evidence accumulation challenge (i.e., accumulating scores in a psychometric model). Using expert ratings on the essence and difficulty of actions in the MBPA, we answer the first challenge by demonstrating that interactive task performance in MBPA can be scored. Furthermore, we answer the second challenge by recoding the expert ratings in conditional probability tables that can be used in a Bayesian Network (a psychometric model for reasoning under uncertainty and complexity). Finally, we validate and illustrate the presented methodology through the analysis of the response data of 57 confined space guard students who performed in the MBPA. Highlights: We present a study on the analysis of performance data of multimedia-based performance assessment. We aim to answer the evidence identification and evidence accumulation challenge in this study. We present data based of 57 students a multimedia-based performance assessment. We use log file analyses. We use Bayes nets to evaluate student performance in the multimedia-based performance assessment.