Visual relationship extraction in images and a semantic interpretation with ontologies. (17th February 2022) - Zga, Adel; Nini, Brahim
Nowadays, three challenges should be considered in order to build a strong model that is used to extract and semantically interpret the relationship between objects in images namely: long-tail problem, large intra-class divergence, and the semantic dependency or semantic gap. In order to overcome those challenges, we propose three main contributions: 1) an ontological semantic model to filter false negatives/positives using a statistical ranking module; 2) a combination of semantic ontological module and visual relationship module that both takes as input the results of the statistical ranking module and produces as output classification of < human - predicate - object >; 3) a semantic model for the visual relationship module that ranks each prediction of relation classes by transferring the spatial relationship onto a high dimension spatial feature. We use HCVRD that highlights two important practical problems, the long-tail distribution issue, and the zero-shot problem. The experimental results on the HCVRD dataset demonstrate the superior performance of the proposed approach.