Statistical semi-supervised system for grading multiple peer-reviewed open-ended works. (November 2018)

• Record Type: Journal Article
• Authors: Rico-Juan, Juan Ramón; Gallego, Antonio-Javier; Valero-Mas, Jose J.; Calvo-Zaragoza, Jorge
• Is Part Of: Computers & education. Volume 126(2018)
• Page Start: 264
• Page End: 282
• Publication Date: 2018-11
• Subjects: Computer-aided assessment -- Automated grading -- Open-ended works; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2018.07.017
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: In the education context, open-ended works generally entail a series of benefits as the possibility of develop original ideas and a more productive learning process to the student rather than closed-answer activities. Nevertheless, such works suppose a significant correction workload to the teacher in contrast to the latter ones that can be self-corrected. Furthermore, such workload turns to be intractable with large groups of students. In order to maintain the advantages of open-ended works with a reasonable amount of correction effort, this article proposes a novel methodology: students perform the corrections using a rubric (closed Likert scale) as a guideline in a peer-review fashion; then, their markings are automatically analyzed with statistical tools to detect possible biased scorings; finally, in the event the statistical analysis detects a biased case, the teacher is required to intervene to manually correct the assignment. This methodology has been tested on two different assignments with two heterogeneous groups of people to assess the robustness and reliability of the proposal. As a result, we obtain values over 95% in the confidence of the intra-class correlation test (ICC) between the grades computed by our proposal and those directly resulting from the manual correction of the teacher. These figures confirm that the evaluation obtained with the proposed methodology is statistically similar to that of the manual correction of the teacher with a remarkable decrease in terms of effort. Highlights: Open-ended works represents a significant teachers' workload with large groups. We propose a novel methodology for open-ended works peer review. Analysis with statistical tools is considered to detect possible biased scorings. We tested the proposal with two different assignments with two groups of students. The proposed methodology is statistically similar to that of the teachers' correction.