Illumination-aware faster R-CNN for robust multispectral pedestrian detection. (January 2019)

• Record Type: Journal Article
• Authors: Li, Chengyang; Song, Dan; Tong, Ruofeng; Tang, Min
• Is Part Of: Pattern recognition. Volume 85(2019:Jan.)
• Page Start: 161
• Page End: 171
• Publication Date: 2019-01
• Subjects: Multispectral pedestrian detection -- Illumination-aware -- Gated fusion; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2018.08.005
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Six convnet fusion architectures and their adaptations are investigated. IAF R-CNN model for multispectral pedestrian detection is proposed. An illumination-aware weighting mechanism is introduced. A new state-of-the-art result on KAIST Benchmark is reported. Abstract: Multispectral images of color-thermal pairs have shown more effective than a single color channel for pedestrian detection, especially under challenging illumination conditions. However, there is still a lack of studies on how to fuse the two modalities effectively. In this paper, we deeply compare six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Further, we discover that pedestrian detection confidences from color or thermal images are correlated with illumination conditions. With this in mind, we propose an Illumination-aware Faster R-CNN (IAF R-CNN). Specifically, an Illumination-aware Network is introduced to give an illumination measure of the input image. Then we adaptively merge color and thermal sub-networks via a gate function defined over the illumination value. The experimental results on KAIST Multispectral Pedestrian Benchmark validate the effectiveness of the proposed IAF R-CNN.