Evaluation of technology use in education: Findings from a critical analysis of systematic literature reviews. (19th December 2019)

• Record Type: Journal Article
• Authors: Lai, Jennifer W. M.; Bower, Matt
• Is Part Of: Journal of computer assisted learning. Volume 36:Number 3(2020)
• Page Start: 241
• Page End: 259
• Publication Date: 2019-12-19
• Subjects: evaluation -- learning technology -- meta‐review -- systematic review -- tertiary review; Computer-assisted instruction -- Periodicals; 371.334
• DOI: 10.1111/jcal.12412
• Languages: English
• ISSNs: 0266-4909

Abstract

Abstract: Because the educational technology research literature is so broad, it is difficult for researchers to acquire an accurate sense of the issues and trends across the entire field. There has not been any recent effort to synthesize or critically analyse the systematic reviews in the area of educational technology. This study analysed 73 systematic literature reviews, including meta‐analyses, focused on the evaluation of technology in educational contexts, in order to holistically understand the impact of learning technology use across different aspects of evaluation. Among these reviews, the most common theme examined was learning outcomes (89%), followed by affective elements (45%), behaviours (25%), technological elements (21%) and teaching/pedagogical aspects (19%). Most of the reviews found that the use of technology improved learning outcomes and affective perceptions. Approaches involving interaction, gamification, constructivism, student‐centred learning and feedback were most effective. The analysis highlighted the need for more reviews focusing on multiple aspects of learning technology evaluation, on school level education and on the use of technology in naturalistic (non‐interventional) settings. Critical reflections are also cast on the methods used to conduct systematic reviews in the educational technology field. Lay Description: What is already known about this topic Systematic literature reviews are often used to understand trends in specific areas of educational technology research and evaluation. There has not been any recent effort to synthesis or critically analyse the reviews in areas of educational technology research. What this paper adds A critical analysis of 73 systematic reviews to understand trends in learning technology evaluation. Technology use consistently improves outcomes across reviews of different learning domains, technologies, and levels. Approaches involving interaction, gamification, constructivism, student‐centred learning and feedback were most effective. The majority of systematic reviews did not evaluate the quality of the primary papers which they examined. Implications of study findings for practitioners Educators should preference approaches that are constructivist, social, gamified, student‐centred and involving feedback. Educators should strive to evaluate the quality of the papers they review as well as multiple aspects of learning. The more detailed findings of this paper will enable educators to adopt a more nuanced approach to their practice.