Dual camera snapshot hyperspectral imaging system via physics-informed learning. (July 2022)

• Record Type: Journal Article
• Authors: Xie, Hui; Zhao, Zhuang; Han, Jing; Zhang, Yi; Bai, Lianfa; Lu, Jun
• Is Part Of: Optics and lasers in engineering. Volume 154(2022)
• Publication Date: 2022-07
• Subjects: Hyperspectral imaging -- Computer vision -- Spectrum -- Neural network -- Deep -- Learning -- Computational imaging; Lasers in engineering -- Periodicals; Optical measurements -- Periodicals; Optics -- Periodicals; Lasers en ingénierie -- Périodiques; Mesures optiques -- Périodiques; Optique -- Périodiques; 621.36605
• DOI: 10.1016/j.optlaseng.2022.107023
• Languages: English
• ISSNs: 0143-8166

Abstract

Highlights: In this manuscript, we consider using the system's optical imaging process with convolutional neural networks (CNNs) to solve the snapshot hyperspectral imaging reconstruction problem, which uses a dual-camera system to capture the three-dimensional hyperspectral images (HSIs) in a compressed way. Specific contributions are: Proposed a physics-informed framework with two self-supervised learning branches. One branch projected the physical process of the CASSI to obtain a compressed image to learn the spectral information. Another branch emulated the imaging process of the color camera with the camera's quantum efficiency to learn the spatial information. Compared with the iterative optimization method, benefit from the physics-informed CNN framework based on the optical imaging process of the color camera and CASSI, the proposed system exhibited excellent reconstruction performance. Besides, after the optimization, the learned model also has good reconstruction ability for the same type scenarios. Which makes it possible to reconstruct scenes in real time. Unlike most supervised deep learning approaches, our method does not require a significant amount of standard data for pre-training. We used the physical imaging process to replace the pathological mapping relationship between the compressive image and the standard HSIs, which resulted in better adaptability to scenes (the situation of no GT). Benefiting from physics-informed self-supervised framework, our method can realize online learning, real-time scene generalization and effective imaging in real scenes. Abstract: We consider using the system's optical imaging process with convolutional neural networks (CNNs) to solve the snapshot hyperspectral imaging reconstruction problem, which uses a dual-camera system to capture the three-dimensional hyperspectral images (HSIs) in a compressed way. Various methods using CNNs have been developed in recent years to reconstruct HSIs, but most of the supervised deep learning methods aimed to fit a brute-force mapping relationship between the captured compressed image and standard HSIs. Thus, the learned mapping would be invalid when the observation data deviate from the training data. Especially, we usually don't have ground truth in real-life scenarios. In this paper, we present a self-supervised dual-camera equipment with an untrained physics-informed CNNs framework. Extensive simulation and experimental results show that our method without training can be adapted to a wide imaging environment with good performance. Furthermore, compared with the training-based methods, our system can be constantly fine-tuned and self-improved in real-life scenarios.