Language-enhanced object reasoning networks for video moment retrieval with text query. (September 2022)

• Record Type: Journal Article
• Authors: Wang, Gongmian; Jiang, Xun; Liu, Ning; Xu, Xing
• Is Part Of: Computers & electrical engineering. Volume 102(2022)
• Publication Date: 2022-09
• Subjects: Video moment retrieval -- Cross-modal retrieval -- Video understanding -- Temporal localization; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2022.108137
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Video moment retrieval with text query aims to retrieve the most relevant video segment from the untrimmed video based on the given text query. Most existing works take advantage of pre-trained video understanding networks, which greatly limit its expressiveness and may ignore the fine-grained interaction between objects in the video. Motivated by this issue, we propose a novel model termed Language-enhanced Object Reasoning Networks (LEORN) which uses the object features extracted by the object detection network and incorporates the language features to model the relation between the objects within the frame. Besides, we design a new temporal shift mechanism to model the temporal relationship of objects while reducing the noise from irrelevant objects in the adjacent frames. The experiments on two challenging datasets, show that our proposed LEORN method with object-level features is a good complement to existing pre-trained video features and outperforms state-of-the-art methods by a simple combination. Highlights: Language-enhanced object reasoning network is used for video moment retrieval. Cross-modal object reasoning module is designed for video object reasoning. Attentive feature query extraction is built to extract object relationships.