Setwise comparison: efficient fine-grained rating of movement videos using algorithmic support – a proof of concept study. Issue 18 (27th August 2020)

• Record Type: Journal Article
• Authors: Steinheimer, Saskia; Dorn, Jonas F.; Morrison, Cecily; Sarkar, Advait; D'Souza, Marcus; Boisvert, Jacques; Bedi, Rishi; Burggraaff, Jessica; Kontschieder, Peter; Dahlke, Frank; Sellen, Abigail; Uitdehaag, Bernard M. J.; Kappos, Ludwig; Kamm, Christian P.
• Is Part Of: Disability and rehabilitation. Volume 42:Issue 18(2020)
• Page Start: 2640
• Page End: 2646
• Publication Date: 2020-08-27
• Subjects: Multiple sclerosis -- motor dysfunction -- disease progression -- setwise comparison -- video rating -- disability; People with disabilities -- Periodicals; Rehabilitation -- Periodicals; 617.03
• DOI: 10.1080/09638288.2018.1563832
• Languages: English
• ISSNs: 0963-8288

Abstract

Abstract: Purpose: Clinical ordinal rating scales of movements, e.g., the Expanded Disability Status Scale, have poor intra- and interrater reliability, are insensitive to subtle differences and result in coarse-grained ratings compared to relative comparative rating methods. We therefore established video-based setwise comparison as a fine-grained, reliable and efficient rating method of motor dysfunction using algorithmic support. Materials and methods: Eight neurologists rated a set of 40 multiple sclerosis patient videos of the Finger-to-Nose-Test using both the newly developed setwise comparison and the established pairwise comparison techniques, which result in a continuous rating scale. Reliability was assessed by the intra-class correlation coefficient. Construct validity was estimated as Pearson's correlation between the continuous scale and severity ratings according to the Neurostatus scale for upper-extremity tremor/dysmetria and the Nine-hole-peg-test. Comparing the time needed for ratings assessed efficiency. Results: Intra-class correlation coefficient was 0.83 for setwise and 0.7 for pairwise comparison. Correlation to the tremor/dysmetria score of the Neurostatus was 0.86 for both rating procedures and correlation to the Nine-hole-peg-test was 0.64 (setwise) and 0.66 (pairwise). The time needed to rate 40 videos was 22.9 ± 6.9 minutes (setwise) and 77.8 ± 14.5 minutes (pairwise). Conclusions: Setwise comparison is an efficient, valid and reliable method for fine-grained rating of motor dysfunction that can be applied to larger datasets. It is substantially more efficient than pairwise comparison. Implications for rehabilitation: Disability rating is crucial in clinical neurorehabilitation and in clinical trials. Humans are naturally inconsistent in rating items on ordinal scales leading to poor intra- and interrater reliability, insensitivity to subtle differences and coarse-grained ratings. Video-based setwise comparison is a new rating method enabling fine-grained, reliable and efficient ratings of motor dysfunction using algorithmic support.