Where to put the image in an image caption generator. (23rd April 2018)
- Record Type:
- Journal Article
- Title:
- Where to put the image in an image caption generator. (23rd April 2018)
- Main Title:
- Where to put the image in an image caption generator
- Authors:
- TANTI, MARC
GATT, ALBERT
CAMILLERI, KENNETH P. - Editors:
- Belz, Anja
Berg, Tamara - Abstract:
- Abstract: When a recurrent neural network (RNN) language model is used for caption generation, the image information can be fed to the neural network either by directly incorporating it in the RNN – conditioning the language model by 'injecting' image features – or in a layer following the RNN – conditioning the language model by 'merging' image features. While both options are attested in the literature, there is as yet no systematic comparison between the two. In this paper, we empirically show that it is not especially detrimental to performance whether one architecture is used or another. The merge architecture does have practical advantages, as conditioning by merging allows the RNN's hidden state vector to shrink in size by up to four times. Our results suggest that the visual and linguistic modalities for caption generation need not be jointly encoded by the RNN as that yields large, memory-intensive models with few tangible advantages in performance; rather, the multimodal integration should be delayed to a subsequent stage.
- Is Part Of:
- Natural language engineering. Volume 24:Part 3(2018)
- Journal:
- Natural language engineering
- Issue:
- Volume 24:Part 3(2018)
- Issue Display:
- Volume 24, Issue 3, Part 3 (2018)
- Year:
- 2018
- Volume:
- 24
- Issue:
- 3
- Part:
- 3
- Issue Sort Value:
- 2018-0024-0003-0003
- Page Start:
- 467
- Page End:
- 489
- Publication Date:
- 2018-04-23
- Subjects:
- Natural language processing (Computer science) -- Periodicals
Software engineering -- Periodicals
006.35 - Journal URLs:
- http://journals.cambridge.org/action/displayJournal?jid=NLE ↗
- DOI:
- 10.1017/S1351324918000098 ↗
- Languages:
- English
- ISSNs:
- 1351-3249
- Deposit Type:
- Legaldeposit
- View Content:
- Available online (eLD content is only available in our Reading Rooms) ↗
- Physical Locations:
- British Library HMNTS - ELD Digital store
- Ingest File:
- 6419.xml