Multimodal object description network for dense captioning. Issue 15 (1st July 2017)

• Record Type: Journal Article
• Authors: Wang, Weixuan; Hu, Haifeng
• Is Part Of: Electronics letters. Volume 53:Issue 15(2017)
• Page Start: 1041
• Page End: 1042
• Publication Date: 2017-07-01
• Subjects: object detection -- computer vision -- feature extraction -- statistical distributions -- visual databases; multimodal object description network -- dense captioning -- MODN model -- vision module -- language module -- R‐CNN -- salient object detection -- feature extraction -- semantics features -- object features -- probability distribution -- multimodal layer -- discriminant information extraction -- VOC2007 dataset -- Visual Genome dataset; Electronics -- Periodicals; 621.381
• DOI: 10.1049/el.2017.0326
• Languages: English
• ISSNs: 0013-5194

Abstract

Abstract : A new multimodal object description network (MODN) model for dense captioning is proposed. The proposed model is constructed by using a vision module and a language module. As for vision module, the modified faster regions‐convolution neural network (R‐CNN) is used to detect the salient objects and extract their inherited features. The language module combines the semantics features with the object features obtained from the vision module and calculate the probability distribution of each word in the sentence. Compared with existing methods, a multimodal layer in the proposed MODN framework is adopted which can effectively extract discriminant information from both object and semantic features. Moreover, MODN can generate object description rapidly without external region proposal. The effectiveness of MODN on the famous VOC2007 dataset and Visual Genome dataset is verified.