Unified multi-spectral pedestrian detection based on probabilistic fusion networks. (August 2018)

• Record Type: Journal Article
• Authors: Park, Kihong; Kim, Seungryong; Sohn, Kwanghoon
• Is Part Of: Pattern recognition. Volume 80(2018:Aug.)
• Page Start: 143
• Page End: 155
• Publication Date: 2018-08
• Subjects: Multi-spectral sensor fusion -- Pedestrian detection -- Channel weighting fusion -- Probabilistic fusion; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2018.03.007
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a unified CNN architecture for the task of multispectral pedestrian detection and formulate the entire network to be learned in an end-to-end manner. Unlike existing multispectral fusion techniques, we comprehensively utilize color, thermal, color-thermal fusion features to maximize detection performance by synergistically using their detection probabilities with channel weighting fusion (CWF) and accumulated probability fusion (APF). The proposed system significantly reduces the missing rate of baseline method by 5.60%, yielding a 31.36% overall missing rate on the KAIST multispectral pedestrian benchmark Abstract: Despite significant progress in machine learning, pedestrian detection in the real-world is still regarded as one of the challenging problems, limited by occluded appearances, cluttered backgrounds, and bad visibility at night. This has caused detection approaches using multi-spectral sensors such as color and thermal which could be complementary to each other. In this paper, we propose a novel sensor fusion framework for detecting pedestrians even in challenging real-world environments. We design a convolutional neural network (CNN) architecture that consists of three-branch detection models taking different modalities as inputs. Unlike existing methods, we consider all detection probabilities from each modality in a unified CNN framework and selectively use them through a channel weighting fusion (CWF) layer to maximize the detection performance. An accumulated probability fusion (APF) layer is also introduced to combine probabilities from different modalities at the proposal-level. We formulate these sub-networks into a unified network, so that it is possible to train the whole network in an end-to-end manner. Our extensive evaluation demonstrates that the proposed method outperforms the state-of-the-art methods on the challenging KAIST, CVC-14, and DIML multi-spectral pedestrian datasets.