A supervised multimodal search re-ranking technique using visual semantics. (24th January 2020)

• Record Type: Journal Article
• Authors: Bhuvan, Nikhila T.; Elayidom, M. Sudheep
• Is Part Of: International journal of intelligent enterprise. Volume 7:Number 1/3(2020)
• Page Start: 279
• Page End: 290
• Publication Date: 2020-01-24
• Subjects: automatic image annotation -- convolutional neural networks -- image descriptor -- multimodality search -- search re ranking -- semantic similarity -- supervised re ranking -- visual semantics; Organizational learning -- Periodicals; Knowledge management -- Periodicals; Business intelligence -- Periodicals; Business enterprises -- Design -- Periodicals; Organizational effectiveness -- Periodicals; 658.005
• Languages: English
• ISSNs: 1745-3232

Abstract

The multimedia content in a webpage is usually given least importance in webpage ranking. A better user satisfaction could be achieved if the web pages are ranked based on multiple modalities rather than just depending on the textual content. A better ranking of the web pages is proposed using natural language descriptions of images along with the textual content in a webpage is being proposed. The inter-modal correspondences between text and visual data are learned using the convolutional neural network assisted by the datasets of images and their sentence descriptors. The model is based on convolutional neural networks over images to generate the image descriptor and Dandelion API for their similarity measure with the query. The image description is algorithmically generated rather depending on the image annotations present. Finally, it has been proven that the re-ranked web pages using the generated descriptions significantly outperform the state of art retrieval models.