Aligning objectives with assessment in online courses: Integrating learning analytics and measurement theory. (December 2022)

• Record Type: Journal Article
• Authors: Barthakur, Abhinava; Joksimovic, Srecko; Kovanovic, Vitomir; Richey, Michael; Pardo, Abelardo
• Is Part Of: Computers & education. Volume 190(2022)
• Publication Date: 2022-12
• Subjects: Educational assessment -- Learning objectives -- Learning analytics -- Multi-dimensional item response theory; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2022.104603
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: Well-designed online courses enhance learning experiences and allow effective development of learners' skills and knowledge. A critical factor contributing to the design of online courses in the higher education settings are well-defined learning objectives that align with course assessments and learning activities. While there are several introspective instruments to evaluate course designs, with the broader adoption of educational technologies and digital tools, there is a wealth of data that offers insights on the alignment of learning objectives to assessments. Such data has paved the way for evidence-based methods of investigating course effectiveness within higher education. This study outlines a methodology for designing and evaluating the alignment between course learning objectives and assessment activities at scale, utilising a combination of learning analytics and measurement theory approaches, more specificially exploratory multi-dimensional item response theory (MIRT) models. We demonstrate the proposed methodology within a professional development MOOC on leadership skills development, where we evaluate the alignemnt between course objectives and reflective writing assessments activities. Our results suggested that the alignment of the existing course objectives to assessment activities can be improved, showing the practical value of the proposed approach. The theoretical and practical implications of this research are further illustrated. Highlights: The alignment of course objectives with assessments is a critical element of online course design in higher education. The design of course objectives and their alignment with assessment should combine theory and data-driven approaches. The assessment data did not support the original alignment developed by the instructors and course designers. The findings from this study demonstrate the importance of aligning assessments with the stated course objectives.