Challenges to assessing motivation in MOOC learners: An application of an argument-based approach. (June 2020)

• Record Type: Journal Article
• Authors: Douglas, Kerrie A.; Merzdorf, Hillary E.; Hicks, Nathan M.; Sarfraz, Muhammad Ihsanulhaq; Bermel, Peter
• Is Part Of: Computers & education. Volume 150(2020)
• Publication Date: 2020-06
• Subjects: validation -- Motivation -- Assessment -- MOOC; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2020.103829
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: While data for examining learner activity are abundant, there are relatively little frameworks for principled interpretation of the behavior. Through application of Kane's argument-based approach to assessment validation, the authors conducted several analyses to combine motivation assessment with online behavioral data for further validating inferences made about MOOC learners. The EVC motivation scale was administered to learners in three advanced engineering MOOCs, and event log data was collected from the online learning platform. The EVC items were comprehensively examined (n = 661) through factor analysis, item response theory, and linear regression. Results indicated that the instrument retained a three-factor structure as well as strict and structural invariance across age groups and education levels, but the Expectancy and Value items suffered from significant ceiling effects, a key difficulty in assessing motivation using this approach. As part of a regression model with learners' intentions, EVC scores did not account for a significant amount of variance in learners' assessment outcomes and course behavior. Challenges remain for adjusting to learners' high expectancy and value of courses at the beginning, and for fully understanding how scores relate to learner behavior. Graphical abstract: Image 1 Highlights: This paper applies an argument-based approach to assessment validation in advanced engineering MOOCs. With learner activity data and motivation scores, we study the validity of the Expectancy-Value-Cost scale for MOOCs. Learners' Expectancy and Value motivation scores were generally high, creating a ceiling effect. Results did not have positive performance predictiveness of Expectancy and Value, or negative predictiveness of Cost.