A critique to Akdemir and Oguz (2008): Methodological and statistical issues to consider when conducting educational experiments. (September 2015)

• Record Type: Journal Article
• Authors: Albers, Casper J.; Boevé, Anja J.; Meijer, Rob R.
• Is Part Of: Computers & education. Volume 87(2015)
• Page Start: 238
• Page End: 242
• Publication Date: 2015-09
• Subjects: Evaluation methodologies -- Methodology in education -- Statistics; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2015.07.001
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: In the paper "Computer-based testing: An alternative for the assessment of Turkish undergraduate students", Akdemir and Oguz (2008) discuss an experiment to compare student performance in paper-and-pencil tests with computer-based tests, and conclude that students taking computer-based tests do not underperform compared to students taking pen-and-pencil tests. In this letter, we indicate two severe methodological and statistical flaws in this paper. We show how, in general, such flaws can affect experimental research. Due to these flaws, the conclusions by Akdemir and Oguz are unfounded: one cannot reach these conclusions on basis of this design and analysis. We provide a set of guidelines and advices to avoid methodological problems when setting up an educational experiment. Highlights: Critical note on the Akdemir and Oguz (2008) study into computer-based test performance. Due to the use of a non-randomized design their results cannot be attributed to test mode. Analyses assuming independent groups were used, while the study used a repeated measure design. These two fundamental flaws render the conclusions of Akdemir and Oguz (2008) unfounded. The paper explains why this is and how to avoid making these mistakes in your own research.