Self-attention driven adversarial similarity learning network. (September 2020)

• Record Type: Journal Article
• Authors: Gao, Xinjian; Zhang, Zhao; Mu, Tingting; Zhang, Xudong; Cui, Chaoran; Wang, Meng
• Is Part Of: Pattern recognition. Volume 105(2020:Sep.)
• Publication Date: 2020-09
• Subjects: Self-attention mechanism -- Adversarial loss -- Similarity learning network -- Explainable deep learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107331
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: To address the problem of conventional similarity learning algorithms that take the entire given objects into account, we take the advantage of the self-attention mechanism to generate self-attention weighted feature maps of given objects and feed them to the further similarity learning step. It ensures the final achieved similarity scores are discriminative because of the semantic information that stored in these selected regions, rather than take the entire object into account. To address the problem of conventional similarity learning algorithms that only aim to distinguish objects and lack semantic interpretability of their obtained similarity score, we propose an interpretable similarity learning method. In addition, we add a generator-discriminator model with adversarial loss to force the topic vectors to capture and preserve the hidden semantic information from the self-attention weighted feature maps of given objects. This is accomplished by propagating the difference between objects generated from topic vectors and real objects to the similarity learning step. In order to obtain global optimized results and prevent the captured semantic information from being wiped during training, we combine the self-attention mechanism, the similarity learning section and the generator-discriminator section together and propose an end-to-end self-attention driven adversarial similarity learning network based on the joint loss function to simultaneously train the above components. Abstract: Similarity learning is a kind of machine learning algorithm that aims to measure the relevance between given objects. However, conventional similarity learning algorithms usually measure the distance between the entire given objects in the latent feature space. Consequently, the obtained similarity scores only represent how close are the entire given objects, but are incapable of demonstrating which part of them are similar to each other and how semantically similar are they. To address the above problems, in this paper, we propose a self-attention driven adversarial similarity learning network. Discriminative self-attention weights are firstly assigned to different regions of the given objects. The similarity learning step measures the relevance between these self-attention weighted feature maps of given objects under various topic vectors. The topic vectors are conditioned to capture and preserve hidden semantic information within data distribution by a generator-discriminator model with adversarial loss. This model aims to generate objects from topic vectors and propagates the difference between the generated and the real objects back to the similarity learning step, which forces the topic vectors to not only assign discriminative similarity scores to different object pairs but also further mine the hidden semantic information within data distribution. The final similarity scores represent how tight the given objects are connected to the topics. In addition, the regions with higher self-attention weights make more contribution to the discriminative similarity scores. The effectiveness of the proposed method is demonstrated through evaluations based on image retrieval task and document retrieval task and compared against various state-of-the-art algorithms in the field. The visualization results of topic vectors and self-attention weighted feature maps are demonstrated to make our proposed method explainable.