22 MotionInput supporting DirectX – a low latency approach to webcam-based navigation of desk-based gestures to interact with electronic health records. (15th December 2021)

• Record Type: Journal Article
• Authors: Almazov, Emil; Visram, Sheena; Sebire, Neil J; Taylor, Andrew; Stott, Lee; Conner, Sue; Molyneux, Gemma; Roberts, Graham; Mohamedally, Dean
• Is Part Of: Archives of disease in childhood. Volume 106(2021)Supplement 3
• Page Start: A8
• Page End: A9
• Publication Date: 2021-12-15
• Subjects: Children -- Diseases -- Periodicals; Infants -- Diseases -- Periodicals; 618.920005
• DOI: 10.1136/archdischild-2021-gosh.22
• Languages: English
• ISSNs: 0003-9888

Abstract

Abstract : Introduction: There have been several attempts to integrate touchless interactions with computer interfaces in operating theatres and interventional radiology. Whilst motivated by infection prevention gains, this type of technology is yet to mainstream in healthcare as depth camera are historically expensive. Here we present MotionInput supporting DirectX for desk gestures. The key idea behind this prototype project is to 'use the tech and tools you already have', to provide touchless interactive interfaces to existing Windows software. Method: This proof-of-concept prototype features Visual Studio based modules that use a regular webcam (e.g., on a laptop) and open-source computer vision libraries to deliver low-latency input on Microsoft Windows 10. The functioning software prototype focuses on several image-processing algorithms, leading to desk-based gestures with in-air pen and gloved hand navigation. Results: Discrete hand motions are tracked via x and y coordinates. These coordinates are then mapped and processed by PyDirectInput functions to replicate movements of the mouse or press keys on the keyboard and click with the mouse. This is extended to recognition of 2 fingers that are gloved, which would apply to healthcare scenarios in which clinicians use one hand to navigate clinical applications including an electronic patient record or scan with one hand, whilst maintaining the second hand as free for other tasks. The prototype has been shown to work with UCL's HoloRepository – an open-source project that enables 3D viewing of CT and MRI DICOM scans of the brain, lungs, chest, abdomen and kidneys. Conclusion: Further research is exploring the application of MotionInput to design touchless interactions with computer interfaces in clinical spaces. Coupled with advances in computer vision, we believe that the convenience of use and ease of access of cameras integrated into existing hardware will improve uptake and help bring gesture recognition software into the mainstream.