Two-stream attention network with local and non-local dependence for referring relationships. (28th March 2022)

• Record Type: Journal Article
• Authors: Hu, Jincheng; Wang, Tao
• Is Part Of: International journal of embedded systems. Volume 15:Number 1(2022)
• Page Start: 53
• Page End: 60
• Publication Date: 2022-03-28
• Subjects: two-stream attention network -- TSAN -- local and non-local dependence -- referring relationships -- cross modal; Embedded computer systems -- Periodicals; 004.16
• Languages: English
• ISSNs: 1741-1068

Abstract

Referring relationships is a task aiming to retrieve entities in the image conditioned on the given structured relationship text. Previous works utilise convolution-network-base model to learn image features conditioned on the text and classify the pixels in the image for retrieval. However, convolution network concentrates on the local dependence in the image without considering correlations between pixels far apart. In this paper, a two-stream attention network (TSAN) is proposed to address the above issue. Firstly, cross-modal feature maps are initialised by adding text information to each pixel in the visual feature maps. Then, TSAN designs attention mechanism for the local and non-local dependence in the cross-modal features to obtain two probability maps. Finally, the network integrates the two probability maps to predict the localisation probability. Experiments are conducted on three datasets and the results demonstrate that the proposed TSAN achieves better performance compared with several powerful models.