Score Comparability between Online Proctored and In‐Person Credentialing Exams. Issue 2 (27th April 2022)

• Record Type: Journal Article
• Authors: Jones, Paul; Tong, Ye; Liu, Jinghua; Borglum, Joshua; Primoli, Vince
• Editors: Moses, Tim; Puhan, Gautam
• Is Part Of: Journal of educational measurement. Volume 59:Issue 2(2022)
• Page Start: 180
• Page End: 207
• Publication Date: 2022-04-27
• Subjects: Educational tests and measurements -- Periodicals; 371.2605
• DOI: 10.1111/jedm.12320
• Languages: English
• ISSNs: 0022-0655

Abstract

Abstract: This article studied two methods to detect mode effects in two credentialing exams. In Study 1, we used a "modal scale comparison approach, " where the same pool of items was calibrated separately, without transformation, within two TC cohorts (TC1 and TC2) and one OP cohort (OP1) matched on their pool‐based scale score distributions. The calibrations from all three groups were used to score the TC2 cohort, designated the validation sample. The TC1 item parameters and TC1‐based thetas and pass rates were more like the native TC2 values than the OP1‐based values, indicating mode effects, but the score and pass/fail decision differences were small. In Study 2, we used a "cross‐modal repeater approach" in which test takers who failed their first attempt in one modality took the test again in either the same or different modality. The two pairs of repeater groups (TC → TC: TC → OP, and OP → OP: OP → TC) were matched exactly on their first attempt scores. Results showed increased pass rate and greater score variability in all conditions involving OP, with mode effects noticeable in both the TC → OP condition and less‐strongly in the OP → TC condition. Limitations of the study and implications for exam developers were discussed.