Implementation fidelity in computerised assessment of book reading. (January 2018)

• Record Type: Journal Article
• Authors: Topping, Keith
• Is Part Of: Computers & education. Volume 116(2018)
• Page Start: 176
• Page End: 190
• Publication Date: 2018-01
• Subjects: Evaluation methodologies -- Gender studies -- Improving classroom teaching -- Pedagogical issues -- Teaching and learning strategies; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2017.09.009
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: Measuring the implementation fidelity (IF) or integrity of interventions is extremely important, since without it a positive or negative outcome cannot be interpreted. However, IF is actually measured relatively rarely. Direct and indirect methods of measurement have been used in the past, but tend to over-emphasize teacher behaviour. This paper focuses on student behaviour collated through computers - an interesting alternative. It deals with the reading of real books and reading achievement, for which variables a very large amount of computerised data was available – on 852, 295 students in 3243 schools. Reading achievement was measured pre-post with STAR Reading, a computerised item-banked adaptive norm-referenced test of reading comprehension. IF came from the Accelerated Reader (AR), which measures understanding of independent reading of real books the student has chosen by a quiz. Results showed higher IF was related to higher achievement. Neither IF nor reading achievement related to socio-economic status. Primary (elementary) schools had higher IF and achievement than secondary (high) schools. Females had higher IF and achievement than males. Students of higher reading ability implemented AR at a higher level, but did not gain in reading at a higher level. However, this computerised method of measuring IF with book reading showed limited reliability, no greater than methods emphasising teacher behaviour. In future, IF measures emphasising student response and those emphasising teacher behaviour need to be blended, although the latter will never generate the sample size of the former. This may be true of implementation fidelity in areas other than book reading. Highlights: Implementation fidelity (IF) important for evaluating outcomes. Focus on computerised assessment of reading of real books (IF) and reading outcomes. Higher IF related to higher outcomes with 852, 295 students. But computerised IF little more reliable than methods emphasising teacher behaviour. In future, both methods should be blended – possibly for IF in areas beyond reading.