Deep learning approach for hyperspectral image demosaicking, spectral correction and high-resolution RGB reconstruction. Issue 4 (4th July 2022)

• Record Type: Journal Article
• Authors: Li, Peichao; Ebner, Michael; Noonan, Philip; Horgan, Conor; Bahl, Anisha; Ourselin, Sébastien; Shapey, Jonathan; Vercauteren, Tom
• Is Part Of: Computer methods in biomechanics and biomedical engineering. Volume 10:Issue 4(2022)
• Page Start: 409
• Page End: 417
• Publication Date: 2022-07-04
• Subjects: Hyperspectral imaging -- demosaicking -- deep learning -- surgical imaging; Imaging systems in biology -- Periodicals; Imaging systems in medicine -- Periodicals; Biomechanics -- Data processing -- Periodicals; Biomedical engineering -- Periodicals; 616.0757
• DOI: 10.1080/21681163.2021.1997646
• Languages: English
• ISSNs: 2168-1163

Abstract

ABSTRACT: Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections. In this work, we propose a supervised learning-based image demosaicking algorithm for snapshot hyperspectral images. Due to the lack of publicly available medical images acquired with snapshot mosaic cameras, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets captured by high-resolution, but slow, hyperspectral imaging devices. Image reconstruction is achieved using convolutional neural networks for hyperspectral image super-resolution, followed by spectral correction using a sensor-specific calibration matrix. The results are evaluated both quantitatively and qualitatively, showing clear improvements in image quality compared to a baseline demosaicking method using linear interpolation. Moreover, the fast processing time of 45 ms of our algorithm to obtain super-resolved RGB or oxygenation saturation maps per image for a state-of-the-art snapshot mosaic camera demonstrates the potential for its seamless integration into real-time surgical hyperspectral imaging applications.