Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching. Issue 3 (3rd April 2019)

• Record Type: Journal Article
• Authors: Kitto, Kirsty; Williams, Cameron; Alderman, Lyn
• Is Part Of: Assessment & evaluation in higher education. Volume 44:Issue 3(2019)
• Page Start: 338
• Page End: 360
• Publication Date: 2019-04-03
• Subjects: student evaluations of teaching (SET) -- distributions -- contextualisation -- hierarchical Bayesian model -- sensemaking; Education, Higher -- Evaluation -- Periodicals; 378
• DOI: 10.1080/02602938.2018.1506909
• Languages: English
• ISSNs: 0260-2938

Abstract

Abstract: Student evaluations of teaching (SETs) have been used to evaluate higher education teaching performance for decades. Reporting SET results often involves the extraction of an average for some set of course metrics, which facilitates the comparison of teaching teams across different organisational units. Here, we draw attention to ongoing problems with the naive application of this approach. Firstly, a specific average value may arise from data that demonstrates very different patterns of student satisfaction. Furthermore, the use of distance measures (e.g. an average) for ordinal data can be contested, and finally, issues of multiplicity increasingly plague approaches using hypothesis testing. It is time to advance the methodology of the field. We demonstrate how multinomial distributions and hierarchical Bayesian methods can be used to contextualise the SET scores of a course to different organisational units and student cohorts, and then show how this approach can be used to extract sensible information about how a distribution is changing.