Finding an optimal balance between agreement and performance in an online reciprocal peer evaluation system. (March 2018)

• Record Type: Journal Article
• Authors: Cho, Kwangsu; Schunn, Christian D.
• Is Part Of: Studies in educational evaluation. Volume 56(2018)
• Page Start: 94
• Page End: 101
• Publication Date: 2018-03
• Subjects: Agreement -- Performance -- Reliability -- Reciprocal peer evaluation -- Maxima strategy -- Inverted U-shaped function; Curriculum evaluation -- Periodicals; Educational tests and measurements -- Periodicals; Programmes d'études -- Evaluation -- Périodiques; Tests et mesures en éducation -- Périodiques; 375.006
• DOI: 10.1016/j.stueduc.2017.12.001
• Languages: English
• ISSNs: 0191-491X

Abstract

Highlights: Reciprocal peer evaluation often assumes that more raters produce better results. The maxima strategy works consistently with agreement perspectives. The maxima strategy has an inverted U-shaped relationship with performance. Abstract: Consistent with movements integrating performance evaluation and improvement, the current study examines an assumption that more peer raters will produce better results or the so-called maxima strategy . This study examines the maxima strategy from both the agreement and performance perspectives with the intent of examining the role of feedback information in reliability and performance through an optimal number of peer raters per student ratee. It was found that the maxima strategy works consistently from agreement perspectives, whereas the relationship between the maxima strategy and student performance improvement follows an inverted U-shaped function. Accordingly, we recommend that the number of peer raters needs to be decided according to the optimal balance between reliability and performance, which maximizes ratees' performances without sacrificing evaluation reliability.