Diverse and styled image captioning using singular value decomposition‐based mixture of recurrent experts. (1st February 2022)

• Record Type: Journal Article
• Authors: Heidari, Marzi; Ghatee, Mehdi; Nickabadi, Ahmad; Pourhasan Nezhad, Arash
• Is Part Of: Concurrency and computation. Volume 34:Number 22(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-02-01
• Subjects: deep learning -- image captioning -- mixture of experts -- singular value decomposition; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6866
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: With significant advances in vision and natural language processing, the generation of image captions becomes a need. Mathews, Xie, and He extended a new model to generate styled captions by separating semantics and style. In continuation of their work, here, a new captioning model is developed, including an image encoder to extract the features, a mixture of recurrent networks to embed the set of extracted features to a group of words, and a sentence generator that combines the obtained words as a stylized sentence. This Mixture of Recurrent Experts (MoRE) system uses a new training algorithm that derives singular value decomposition from weighting matrices of Recurrent Neural Networks (RNNs) to increase the diversity of captions. Each decomposition step depends on a distinctive factor based on the number of RNNs in MoRE. The used sentence generator gives a stylized language corpus without paired images. Besides, the styled and diverse captions are extracted without training on a densely labeled or styled dataset. MoRE on the COCO dataset generated diverse and stylized image captions without the necessity of extra‐labeling and improved descriptions in terms of content accuracy.