The development of a video retrieval system using a clinician-led approach. (15th March 2020)

• Record Type: Journal Article
• Authors: Darby, John; Sánchez, María B.; Bew, Sarah; Loram, Ian; Butler, Penelope
• Is Part Of: Expert systems with applications. Volume 142(2020)
• Publication Date: 2020-03-15
• Subjects: Search methods -- Telemedicine -- Computer vision -- Videoconferences -- Video sharing; 00-01 -- 99-00; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2019.112992
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: A search tool lets clinicians analyse head movements in video recorded at home. Query-by-movement approach allows clinicians to find specific/new movements easily. High rate of identification of relevant movements with significant time savings. A clinician-led approach to help overcome the perception of technology push. Potential benefits for patient outcomes, therapist time and reduction in cost. Abstract: Patient video taken at home can provide valuable insights into the recovery progress during a programme of physical therapy, but is very time consuming for clinician review. Our work focussed on (i) enabling any patient to share information about progress at home, simply by sharing video and (ii) building intelligent systems to support Physical Therapists (PTs) in reviewing this video data and extracting the necessary detail. This paper reports the development of the system, appropriate for future clinical use without reliance on a technical team, and the clinician involvement in that development. We contribute an interactive content-based video retrieval system that significantly reduces the time taken for clinicians to review videos, using human head movement as an example. The system supports query-by-movement (clinicians move their own body to define search queries) and retrieves the essential fine-grained movements needed for clinical interpretation. This is done by comparing sequences of image-based pose estimates (here head rotations) through a distance metric (here Fréchet distance) and presenting a ranked list of similar movements to clinicians for review. In contrast to existing intelligent systems for retrospective review of human movement, the system supports a flexible analysis where clinicians can look for any movement that interests them. Evaluation by a group of PTs with expertise in training movement control showed that 96% of all relevant movements were identified with time savings of as much as 99.1% compared to reviewing target videos in full. The novelty of this contribution includes retrospective progress monitoring that preserves context through video, and content-based video retrieval that supports both fine-grained human actions and query-by-movement. Future research, including large clinician-led studies, will refine the technical aspects and explore the benefits in terms of patient outcomes, PT time, and financial savings over the course of a programme of therapy. It is anticipated that this clinician-led approach will mitigate the reported slow clinical uptake of technology with resulting patient benefit.