Recurrent bag-of-features for visual information analysis. (October 2020)

• Record Type: Journal Article
• Authors: Krestenitis, Marios; Passalis, Nikolaos; Iosifidis, Alexandros; Gabbouj, Moncef; Tefas, Anastasios
• Is Part Of: Pattern recognition. Volume 106(2020:Oct.)
• Publication Date: 2020-10
• Subjects: Bag-of-Features -- Recurrent neural networks -- Pooling operators -- Activity recognition; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107380
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Additional experiments were conducted with one state-of-the-art pooling method to further highlight the effectiveness of the ReBoF method. Additional experiments were conducted to evaluate the stability of the proposed method for different number of codewords. Conducted additional experiments to provide quantitative results of the proposed method (both for video and image applications). Additional figures were included to better explain how the proposed method works and how it can be applied on different scenarios. Abstract: Deep Learning (DL) has provided powerful tools for visual information analysis. For example, Convolutional Neural Networks (CNNs) are excelling in complex and challenging image analysis tasks by extracting meaningful feature vectors with high discriminative power. However, these powerful feature vectors are crushed through the pooling layers of the network, that usually implement the pooling operation in a less sophisticated manner. This can lead to significant information loss, especially in cases where the informative content of the data is sequentially distributed over the spatial or temporal dimension, e.g., videos, which often require extracting fine-grained temporal information. A novel stateful recurrent pooling approach, that can overcome the aforementioned limitations, is proposed in this paper. The proposed method is inspired by the well-known Bag-of-Features (BoF) model, but employs a stateful trainable recurrent quantizer, instead of plain static quantization, allowing for efficiently processing sequential data and encoding both their temporal, as well as their spatial aspects. The effectiveness of the proposed Recurrent BoF model to enclose spatio-temporal information compared to other competitive methods is demonstrated using six different datasets and two different tasks.