More efficient processes for creating automated essay scoring frameworks: A demonstration of two algorithms. (April 2021)

• Record Type: Journal Article
• Authors: Shin, Jinnie; Gierl, Mark J.
• Is Part Of: Language testing. Volume 38:Number 2(2021)
• Page Start: 247
• Page End: 272
• Publication Date: 2021-04
• Subjects: Automated essay scoring -- Coh-Metrix -- complex language features -- convolutional neural networks -- deep-neural; Language and languages -- Ability testing -- Periodicals; Language and languages -- Examinations -- Periodicals; 407.6
• DOI: 10.1177/0265532220937830
• Languages: English
• ISSNs: 0265-5322

Abstract

Automated essay scoring (AES) has emerged as a secondary or as a sole marker for many high-stakes educational assessments, in native and non-native testing, owing to remarkable advances in feature engineering using natural language processing, machine learning, and deep-neural algorithms. The purpose of this study is to compare the effectiveness and the performance of two AES frameworks, each based on machine learning with deep language features, or complex language features, and deep neural algorithms. More specifically, support vector machines (SVMs) in conjunction with Coh-Metrix features were used for a traditional AES model development, and the convolutional neural networks (CNNs) approach was used for more contemporary deep-neural model development. Then, the strengths and weaknesses of the traditional and contemporary models under different circumstances (e.g., types of the rubric, length of the essay, and the essay type) were tested. The results were evaluated using the quadratic weighted kappa (QWK) score and compared with the agreement between the human raters. The results indicated that the CNNs model performs better, meaning that it produced more comparable results to the human raters than the Coh-Metrix + SVMs model. Moreover, the CNNs model also achieved state-of-the-art performance in most of the essay sets with a high average QWK score.