Visual relationship extraction in images and a semantic interpretation with ontologies. (17th February 2022)

• Record Type: Journal Article
• Authors: Zga, Adel; Nini, Brahim
• Is Part Of: International journal of intelligent information and database systems. Volume 15:Number 2(2022)
• Page Start: 223
• Page End: 247
• Publication Date: 2022-02-17
• Subjects: deep learning -- semantic gap -- ontologies -- human-object interaction -- large intra-class divergence; Database management -- Computer programs -- Periodicals; Information retrieval -- Computer programs -- Periodicals; Information storage and retrieval systems -- Computer programs -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Intelligent agents (Computer software) -- Periodicals; 006.33
• Languages: English
• ISSNs: 1751-5858

Abstract

Nowadays, three challenges should be considered in order to build a strong model that is used to extract and semantically interpret the relationship between objects in images namely: long-tail problem, large intra-class divergence, and the semantic dependency or semantic gap. In order to overcome those challenges, we propose three main contributions: 1) an ontological semantic model to filter false negatives/positives using a statistical ranking module; 2) a combination of semantic ontological module and visual relationship module that both takes as input the results of the statistical ranking module and produces as output classification of < human - predicate - object >; 3) a semantic model for the visual relationship module that ranks each prediction of relation classes by transferring the spatial relationship onto a high dimension spatial feature. We use HCVRD that highlights two important practical problems, the long-tail distribution issue, and the zero-shot problem. The experimental results on the HCVRD dataset demonstrate the superior performance of the proposed approach.