Spike detection: Inter-reader agreement and a statistical Turing test on a large data set. Issue 1 (January 2017)

• Record Type: Journal Article
• Authors: Scheuer, Mark L.; Bagic, Anto; Wilson, Scott B.
• Is Part Of: Clinical neurophysiology. Volume 128:Issue 1(2017:Jan.)
• Page Start: 243
• Page End: 250
• Publication Date: 2017-01
• Subjects: EEG -- Automated spike detection -- Epileptiform -- Artificial neural network -- Noninferiority -- Inter-reader agreement; Neurophysiology -- Periodicals; Electroencephalography -- Periodicals; Electromyography -- Periodicals; Neurology -- Periodicals; 612.8
• DOI: 10.1016/j.clinph.2016.11.005
• Languages: English
• ISSNs: 1388-2457

Abstract

Highlights: Inter-reader and algorithm versus human agreement for spike detection using pairwise comparisons. A statistical Turing test evaluates for algorithm noninferiority versus skilled human performance. The Persyst 13 spike detection algorithm proved noninferior to a set of three skilled human readers. Abstract: Objective: Compare the spike detection performance of three skilled humans and three computer algorithms. Methods: 40 prolonged EEGs, 35 containing reported spikes, were evaluated. Spikes and sharp waves were marked by the humans and algorithms. Pairwise sensitivity and false positive rates were calculated for each human–human and algorithm-human pair. Differences in human pairwise performance were calculated and compared to the range of algorithm versus human performance differences as a type of statistical Turing test. Results: 5474 individual spike events were marked by the humans. Mean, pairwise human sensitivities and false positive rates were 40.0%, 42.1%, and 51.5%, and 0.80, 0.97, and 1.99/min. Only the Persyst 13 (P13) algorithm was comparable to humans – 43.9% and 1.65/min. Evaluation of pairwise differences in sensitivity and false positive rate demonstrated that P13 met statistical noninferiority criteria compared to the humans. Conclusion: Humans had only a fair level of agreement in spike marking. The P13 algorithm was statistically noninferior to the humans. Significance: This was the first time that a spike detection algorithm and humans performed similarly. The performance comparison methodology utilized here is generally applicable to problems in which skilled human performance is the desired standard and no external gold standard exists.