Do online readiness surveys do what they claim? Validity, reliability, and subsequent student enrollment decisions. (July 2016)

• Record Type: Journal Article
• Authors: Wladis, Claire; Samuels, Jason
• Is Part Of: Computers & education. Volume 98(2016)
• Page Start: 39
• Page End: 56
• Publication Date: 2016-07
• Subjects: Online readiness survey -- Online learning -- Retention -- Predictive validity -- Reliability; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2016.03.001
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: Online readiness surveys are commonly administered to students who wish to enroll in online courses in college. However, there have been no well-controlled studies to confirm whether these instruments predict online outcomes specifically (as opposed to predicting course outcomes more generally). This study used a sample of 24, 006 students to test the validity and reliability of an online readiness survey similar to those used in practice at a majority of U.S. colleges. Multilevel models were used to determine if it was a valid predictor of differential online versus face-to-face course outcomes while controlling for unobserved heterogeneity among courses taken by the same student. Student self-selection into online courses was also controlled using student-level covariates. The study also tested the extent to which survey score correlated with subsequent decisions to enroll in an online course. No aspect of the survey was a significant predictor of differential online versus face-to-face performance. In fact, student characteristics commonly collected by institutional research departments were better predictors of differential online versus face-to-face course outcomes than the survey. Furthermore, survey score was inversely related to subsequent online enrollment rates, suggesting that the use of online readiness surveys may discourage some students from enrolling in online courses even when they are not at elevated risk online. This suggests that institutions should be extremely cautious about implementing online readiness surveys before they have been rigorously tested for validity in predicting differential online versus face-to-face outcomes. Highlights: Online readiness surveys as currently implemented may have no predictive validity. Student characteristics may identify students at-risk online better than surveys. Readiness surveys may discourage the wrong students from enrolling online. Colleges should test readiness surveys for predictive validity before implementation. Colleges may want to consider alternatives to online readiness surveys.