Coordinate CNNs and LSTMs to categorize scene images with multi-views and multi-levels of abstraction. (15th April 2019)

• Record Type: Journal Article
• Authors: Bai, Shuang; Tang, Huadong; An, Shan
• Is Part Of: Expert systems with applications. Volume 120(2019)
• Page Start: 298
• Page End: 309
• Publication Date: 2019-04-15
• Subjects: Scene categorization -- Multi-views -- Multi-levels of abstraction -- Long short-term memory -- Convolutional neural networks; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2018.08.056
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: Scene categorization is performed with multi-views and multi-levels of abstraction. CNNs and LSTMs are coordinated to categorize scene images. CNNs are used to generate features of multi-levels of abstraction. LSTMs are used to accommodate multiple image views. Abstract: Due to complexities of scene images, scene categorization is a challenging task in the computer vision community. To categorize scene images effectively, in this paper, we propose to coordinate Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs) to perform scene categorization with multi-views and multi-levels of abstraction. Specifically, to utilize the complementary properties of features of different levels of abstraction, we employ CNNs to extract features of multi-levels of abstraction based on its hierarchical structure. Furthermore, in order to deal with variations in scene image contents, we represent each image with multiple views, and in order to take correlation between image views into consideration, we treat image view features from the same image as a sequence and employ Long Short-Term Memory networks (LSTMs) to perform classification. Based on the proposed method, information of multi-views and multi-levels of abstraction can be made full use of in a single framework. We evaluate the proposed method on two challenging scene datasets, MIT indoor scene 67 and SUN 397. Obtained results demonstrate the effectiveness of utilizing CNNs and LSTMs to categorize scene images with multi-views and multi-levels of abstraction. Experiments on comparison to state-of-the-art methods show that the proposed method outperforms all the other methods used for comparison.