Understanding visual scenes. (28th March 2018)

• Record Type: Journal Article
• Authors: SILBERER, CARINA; UIJLINGS, JASPER; LAPATA, MIRELLA
• Editors: Belz, Anja; Berg, Tamara
• Is Part Of: Natural language engineering. Volume 24:Part 3(2018)
• Page Start: 441
• Page End: 465
• Publication Date: 2018-03-28
• Subjects: Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324918000104
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: A growing body of recent work focuses on the challenging problem of scene understanding using a variety of cross-modal methods which fuse techniques from image and text processing. In this paper, we develop representations for the semantics of scenes by explicitly encoding the objects detected in them and their spatial relations. We represent image content via two well-known types of tree representations, namely constituents and dependencies. Our representations are created deterministically, can be applied to any image dataset irrespective of the task at hand, and are amenable to standard NLP tools developed for tree-based structures. We show that we can apply syntax-based SMT and tree kernel methods in order to build models for image description generation and image-based retrieval. Experimental results on real-world images demonstrate the effectiveness of the framework.