Infrared and visible image fusion using a shallow CNN and structural similarity constraint. Issue 14 (12th October 2020)

• Record Type: Journal Article
• Authors: Li, Lei; Xia, Zhaoqiang; Han, Huijian; He, Guiqing; Roli, Fabio; Feng, Xiaoyi
• Is Part Of: IET image processing. Volume 14:Issue 14(2020)
• Page Start: 3562
• Page End: 3571
• Publication Date: 2020-10-12
• Subjects: learning (artificial intelligence) -- infrared imaging -- image fusion -- convolutional neural nets; convolutional layer -- fused single image -- visible image -- structural similarity constraint -- infrared image fusion method -- training data -- scarce reference images -- multisource images -- deep models -- end‐to‐end shallow convolutional neural network -- visible image fusion -- shallow CNN -- rectified linear unit function -- structural similarity loss -- pixel misalignment; Image processing -- Periodicals; 621.36705
• DOI: 10.1049/iet-ipr.2020.0360
• Languages: English
• ISSNs: 1751-9659

Abstract

Abstract : In recent years, image fusion methods based on deep networks have been proposed to combine infrared and visible images for achieving better fusion image. However, issues such as limited training data, scarce reference images and misalignment of multi‐source images, still limit the fusion performance. To address these problems, we propose an end‐to‐end shallow convolutional neural network with structural constraints, which has only one convolutional layer to fuse infrared and visible images. Different from other methods, our proposed model requires less training data and reference images and is more robust to the misalignment of a couple of images. More specifically, the infrared image and the visible image are first provided as inputs to a convolutional layer to extract the information that should be fused; then, all feature maps are concatenated together and fed into a convolutional layer with one channel to obtain the fused image; finally, a structural similarity loss between the fused image and the input infrared and visible images is computed to update the network parameters and eliminate the effects of pixel misalignment. Extensive experiments show the effectiveness of our proposed method on fusion of infrared and visible images with the performance that outperforms the state‐of‐the‐art methods.