A review of the meta‐analysis by Tingir and colleagues (2017) on the effects of mobile devices on learning. (3rd May 2021)

• Record Type: Journal Article
• Authors: Bissonnette, Steve; Boyer, Christian
• Is Part Of: Journal of computer assisted learning. Volume 38:Number 1(2022)
• Page Start: 1
• Page End: 5
• Publication Date: 2021-05-03
• Subjects: effect -- meta‐analysis -- methodological flaws -- mobile devices; Computer-assisted instruction -- Periodicals; 371.334
• DOI: 10.1111/jcal.12557
• Languages: English
• ISSNs: 0266-4909

Abstract

Abstract: Tingir et al. (2017) concluded from their meta‐analysis that the subject areas taught through mobile devices had significantly higher achievement scores ( d = 0.48) than the ones taught with traditional teaching methods. Given the relatively high positive effect of mobile devices on student achievement, we carefully analysed the selected research in this meta‐analysis. We reviewed Tingir et al.'s (2017) meta‐analysis based on analysis of the methodology of the selected research, while drawing on the work of Slavin (2003), Cheung and Slavin (2016), and Sung et al. (2019). Twelve of the 14 (86%) studies included in the meta‐analysis done by Tingir and his team (2017) present such major methodological flaws that they should not have been included. Our analysis leads us to believe that the conclusion of Tingir et al. (2017) is not justified. It is recognized that duration of experiment is negatively correlated with effect size: the shorter the duration, the higher the effect (Burston, 2015; Slavin & Lake, 2009). Although demanding more effort, the field of education must raise the bar if it is to have knowledge of acceptable value. Lay Description: What is already known about this topic?: Tingir et al. (2017) concluded from their meta‐analysis that: "the subject areas taught through mobile devices had significantly higher achievement scores than the ones taught with traditional teaching methods" (p. 366). What this paper adds?: We therefore reviewed the meta‐analysis by Tingir et al. (2017) and the underlying research. Our analysis leads us to believe that the conclusion of Tingir et al. (2017) is not justified. The data on reading used in the meta‐analysis by Tingir et al. (2017) do not make it possible to calculate a legitimate effect size for reading or to make a judgment about the effectiveness of mobile devices in the classroom. With respect to mathematics, the weakness and biases of the studies selected in the meta‐analysis by Tingir et al. (2017) should not have allowed them to calculate an effect size and do not provide a clear answer on the effectiveness of mobiles devices in the classroom. The studies in science selected by Tingir et al. (2017) demonstrate a set of methodological flaws that cannot provide a sound basis for determining a convincing effect of mobile device in the classroom on student achievement. Implications for practice and/or policy: Tingir and associates (2017) calculated an overall mean effect of 0.48 for mobiles devices on student achievement by combining research in reading (effect = 0.666; N = 3), mathematics (effect = 0.16; N = 3) and science (effect = 0.528; N = 8). The authors did not do a school‐level analysis (elementary, high school) in their school subject groupings due to the small number of studies, particularly in reading and mathematics. Twelve of the 14 (86%) studies included in the meta‐analysis done by Tingir and his team (2017) present such major methodological flaws that they should not have been included. As mentioned in the introduction, Slavin and his colleagues found different results in a related field. Slavin et al. (2009) found an effect size of 0.11 for the use of technology in the classroom, in computer‐assisted and computer‐based instruction, and an effect size of 0.05 for the use of digital tools 10 years later (Slavin, 2019). Given all the above, it is our opinion that the meta‐analysis by Tingir et al. (2017) cannot be used to justify the use of mobile devices in the classroom.