Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training. (August 2018)

• Record Type: Journal Article
• Authors: Westera, Wim; Dascalu, Mihai; Kurvers, Hub; Ruseti, Stefan; Trausan-Matu, Stefan
• Is Part Of: Computers & education. Volume 123(2018)
• Page Start: 212
• Page End: 224
• Publication Date: 2018-08
• Subjects: Intelligent tutoring systems -- Architectures for educational technology system -- Interactive learning environments -- Simulations -- Distance education and telelearning; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2018.05.010
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: This paper presents a methodology for applying automated essay scoring in educational settings. The methodology was tested and validated on a dataset of 173 reports (in Dutch language) that students have created in an applied game on environmental policy. Natural Language Processing technologies from the ReaderBench framework were used to generate an extensive set of textual complexity indices for each of the reports. Afterwards, different machine learning algorithms were used to predict the scores. By combining binary classification (pass or fail) and a probabilistic model for Precision, a trade-off can be made between validity of automated score prediction (Precision) and the reduction of teacher workload required for manual assessment. It was found from the sample that substantial workload reduction can be achieved, while preserving high precision: allowing for a precision of 95% or higher would already reduce the teacher's workload to 74%; lowering precision to 80% produces a workload reduction of 50%. Highlights: We present a methodology that makes automated essay grading results more reliable. It uses a binary classification (pass or fail) and a probabilistic model for precision. It produces an estimate of the reliability of each predicted score separately. Substantial tutor workload reduction is achieved, while preserving high precision. An explicit trade-off between workload reduction and precision can now be made.