Absence of evidence is not evidence of absence. On the limited use of regression discontinuity analysis in higher education. Issue 1 (2nd January 2023)

• Record Type: Journal Article
• Authors: van Dorresteijn, Chevy; Kan, Kees-Jan; Smits, Niels
• Is Part Of: Assessment & evaluation in higher education. Volume 48:Issue 1(2023)
• Page Start: 16
• Page End: 26
• Publication Date: 2023-01-02
• Subjects: regression discontinuity analysis -- simulation study -- statistical power -- resit effect; Education, Higher -- Evaluation -- Periodicals; 378
• DOI: 10.1080/02602938.2021.2016606
• Languages: English
• ISSNs: 0260-2938

Abstract

Abstract: When higher education students are assessed multiple times, teachers need to consider how these assessments can be combined into a single pass or fail decision. A common question that arises is whether students should be allowed to take a resit. Previous research has found little to no clear learning benefits of resits and therefore suggested they might not be advantangeous as they are costly for both students and institutions. However, we conducted a simulation study that shows such a conclusion to be presumptuous. Absence of evidence is not evidence of absence; our results illustrate that if a resit effect were to exist, the analysis used in these studies (i.e. regression discontinuity analysis; RDA) lacked the power to detect such an effect. Power of RDA was only sufficient under extremely implausible conditions (i.e. large sample, large effect size, high correlation between examinations). To adequately compare the effect of assessment policies, researchers are recommended to use other methods than RDA.