Iterative brain tumor retrieval for MR images based on user's intention model. (July 2022)

• Record Type: Journal Article
• Authors: Sun, Mengli; Zou, Wei; Hu, Nan; Wang, Jiajun; Chi, Zheru
• Is Part Of: Pattern recognition. Volume 127(2022)
• Publication Date: 2022-07
• Subjects: CBIR -- Brain tumor images -- Eye-tracking -- Intention similarity -- Iterative retrieval -- Relevance feedback; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.108650
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Involve the physician in the retrieval loop ("user-in-the-loop"). Model the intention of the user based on the eye-tracking data of him/her collected when inspecting the relevance between different images. Define similarity based both on the intention model and image visual features. Use visual features as a bridge to transfer eye movement data. Continuously improve retrieval performance through iterative feedback. Abstract: Generally, medical content-based image retrieval (CBIR) systems select low-level visual features as image descriptors. However, these descriptors fail to provide clues for understanding the content of medical images in a similar way as a human expert, which makes the retrieval results inconsistent with the user's intention. To solve this problem, we propose a closed-loop brain tumor retrieval system for MR images with an eye-tracking based relevance feedback mechanism. In our method, we first model the intention of the user by training a convolutional neural network based on the temporal and spatial features extracted from his/her eye-tracking data collected when inspecting the relevance between different images. Upon using visual features as a bridge, the relevancy degree to the query image of any of the database images is computed with our user's intention model by transferring to it the eye movement data from the most visually similar image amongst images iteratively accumulated in the canvas. Our proposed retrieval system is implemented in an iterative manner. In each round of iteration, user's eye movement data when inspecting the system returns are collected and the canvas collection of images is also updated by appending to it the user inspected system returns. With the updated canvas collections, the relevancy degree of database images can be recomputed and the system can begin a new round search of the most relevant images. Extensive experiments have been performed on a publicly available T1-weighted contrast-enhanced magnetic resonance image (CE-MRI) dataset that consists of three types of brain tumors (glioma, meningioma, and pituitary tumor) collected from 233 patients with a total of 3064 images across the axial, coronal, and sagittal views. Experimental results of 22 volunteers (11 males and 11 females, with an average age of 24.4 years) from our medical school show that upon implicit involvement of users in the brain tumor retrieving process, our proposed system significantly outperforms state-of-the-art methods and achieves P r e c @ 10 to 99.94 %, m A P to 97.95 % after the third round of iteration.