Category co-occurrence modeling for large scale scene recognition. (November 2016)

• Record Type: Journal Article
• Authors: Song, Xinhang; Jiang, Shuqiang; Herranz, Luis; Kong, Yan; Zheng, Kai
• Is Part Of: Pattern recognition. Volume 59(2016:Nov.)
• Page Start: 98
• Page End: 111
• Publication Date: 2016-11
• Subjects: Scene recognition -- Co-occurrence modeling -- Semantic space -- Feature embedding -- Multiple feature combination -- Large scale image recognition; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2016.01.019
• Languages: English
• ISSNs: 0031-3203

Abstract

Abstract: Scene recognition involves complex reasoning from low-level local features to high-level scene categories. The large semantic gap motivates that most methods model scenes resorting to mid-level representations (e.g. objects, topics). However, this implies an additional mid-level vocabulary and has implications in training and inference. In contrast, the semantic multinomial (SMN) represents patches directly in the scene-level semantic space, which leads to ambiguity when aggregated to a global image representation. Fortunately, this ambiguity appears in the form of scene category co-occurrences which can be modeled a posteriori with a classifier. In this paper we observe that these patterns are essentially local rather than global, sparse, and consistent across SMNs obtained from multiple visual features. We propose a co-occurrence modeling framework where we exploit all these patterns jointly in a common semantic space, combining both supervised and unsupervised learning. Based on this framework we can integrate multiple features and design embeddings for large scale recognition directly in the scene-level space. Finally, we use the co-occurrence modeling framework to develop new scene representations, which experiments show that outperform previous SMN-based representations. Abstract : Highlights: A general framework for modeling scene category co-occurrences. Multiple features integration and unsupervised filtering for co-occurrences modeling. Semantic representations (filtered SMNs, co-codes and KCNF embedding).