A CNN-based multi-task framework for weather recognition with multi-scale weather cues. (15th July 2022)

• Record Type: Journal Article
• Authors: Xie, Kezhen; Huang, Lei; Zhang, Wenfeng; Qin, Qibing; Lyu, Lei
• Is Part Of: Expert systems with applications. Volume 198(2022)
• Publication Date: 2022-07-15
• Subjects: Multi-task weather recognition -- Multi-scale weather-cue features -- Weather-cue segmentation -- Weather classification; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.116689
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Automatic image-based weather recognition has a great significance in the practical use of computer visual applications. The weather recognition approaches via multi-task learning obtain the description of weather conditions by jointly tackling weather-cue segmentation task and weather classification task. However, the existing approaches neglect the multiple scales of weather cues, which may decrease the ability to deal with both large and small weather cues in weather-cue segmentation. They also fail to assign appropriate weights, which the multi-task learning for weather recognition relies on, to the two tasks. To address these problems, we propose a novel end-to-end Convolutional Neural Network (CNN)-based Multi-task Weather Recognition Network with multi-scale weather cues (CMWRN). Specifically, in the weather-cue segmentation task, we propose to segment weather cues, e.g. blue sky and black clouds, by capturing multi-scale weather-cue features. Arbitrary-scale weather-cue regions in the outdoor images can be accurately and effectively classified via resampling the weather-cue features at multiple scales. In the weather classification task, a generalized pooling method is introduced to generate more discriminative weather representations. To further promote the weather classification, the multi-scale weather-cue features are exploited to provide weather clues for weather representations. Moreover, we develop an adaptive weighting scheme to automatically balance the weather classification task and weather-cue segmentation task for more effective training. Compared with the state-of-the-arts, experimental results on two public available benchmark datasets demonstrate the superior performance of our approach. Highlights: We capture multi-scale weather-cue features for multi-task weather recognition. A generalized pooling is proposed to generate weather feature representations. An adaptive weighting scheme is proposed for multi-task weather recognition. The state-of-the-art results are reported on two weather recognition datasets.