The role of image representations in vision to language tasks. (21st March 2018)

• Record Type: Journal Article
• Authors: MADHYASTHA, PRANAVA; WANG, JOSIAH; SPECIA, LUCIA
• Editors: Belz, Anja; Berg, Tamara
• Is Part Of: Natural language engineering. Volume 24:Part 3(2018)
• Page Start: 415
• Page End: 439
• Publication Date: 2018-03-21
• Subjects: Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324918000116
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: Tasks that require modeling of both language and visual information, such as image captioning, have become very popular in recent years. Most state-of-the-art approaches make use of image representations obtained from a deep neural network, which are used to generate language information in a variety of ways with end-to-end neural-network-based models. However, it is not clear how different image representations contribute to language generation tasks. In this paper, we probe the representational contribution of the image features in an end-to-end neural modeling framework and study the properties of different types of image representations. We focus on two popular vision to language problems: The task of image captioning and the task of multimodal machine translation. Our analysis provides interesting insights into the representational properties and suggests that end-to-end approaches implicitly learn a visual-semantic subspace and exploit the subspace to generate captions.