Detecting cheating in written medical examinations by statistical analysis of similarity of answers: pilot study. Issue 7499 (5th May 2005)

• Record Type: Journal Article
• Authors: McManus, I C; Lissauer, Tom; Williams, S E
• Is Part Of: BMJ. Volume 330:Issue 7499(2005)
• Page Start: 1064
• Page End: 1066
• Publication Date: 2005-05-05
• Subjects: Medicine -- Periodicals; Medicine -- Periodicals; Medicine; Periodicals; 610
• DOI: 10.1136/bmj.330.7499.1064
• Languages: English
• ISSNs: 0007-1447

Abstract

Abstract: Objective To assess whether a computer program using a variant of Angoff's method can detect anomalous behaviour indicative of cheating in multiple choice medical examinations. Design Statistical analysis of 11 examinations held by the Royal College of Paediatrics and Child Health. Setting UK postgraduate medical examination. Participants Examination candidates. Main outcome measures Detection of anomalous candidate pairs by regression of similarity of correct answers in all possible pairs of candidates on the overall proportion of correct answers. Anomalous pairs were subsequently assessed in terms of examination centres and the seating plan of candidates, to assess adjacency. Results The 11 examinations were taken by a total of 11 518 candidates, and Acinonyx examined 6 178 628 pairs of candidates. Two examinations showed no anomalies, and one examination found an anomaly resulting from a scanning error. The other eight examinations showed 13 anomalies compatible with cheating, and in each pair the two candidates had sat the examination at the same centre, and for six examinations with seating plans, the candidates in the anomalous pairs had been seated side by side. The raw probabilities of the anomalies varied from 3.9x10 -11 to 9.3x10 -30 (median = 1.1x10 -17 ), with Bonferroni-corrected probabilities in the range 2.4x10 -5 to 4.1x10 -24 (median = 1.6x10 -11 ). This suggests that one anomalous pair is found for every 1000 or so candidates taking this postgraduate examination. Conclusions This statistical technique identified a small proportion of candidates who had very similar patterns of correctly answered questions. The likelihood is that one candidate has copied from the other, or that there was collusion, or that a technical error occurred in the exams department (as happened in a single case). Analysis of similarities can be used to identify cheating and as part of the quality assurance process of postgraduate medical examinations.