Gaze prediction based on long short-term memory convolution with associated features of video frames. (April 2023)

• Record Type: Journal Article
• Authors: Xiao, Limei; Zhu, Zizhong; Liu, Hao; Li, Ce; Fu, Wenhao
• Is Part Of: Computers & electrical engineering. Volume 107(2023)
• Publication Date: 2023-04
• Subjects: Gaze prediction -- Dynamic video -- Central prior knowledge -- Convolutional LSTM; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2023.108625
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Gaze prediction is a key issue for visual perception research. It can be used to infer important regions in videos to reduce the amount of computation in learning and inference of various analysis tasks. Vanilla methods for dynamic video unable to extract valid features, and the motion information among dynamic video frames are ignored, which lead to poor prediction results. We propose a gaze prediction based on LSTM convolution with associated features of video frames (LSTM-CVFAF). Firstly, by adding learnable central prior knowledge, the proposed method can effectively and accurately extract the spatial information of each frame. Secondly, the LSTM is deployed to get temporal motion gaze features. Finally, the spatial and temporal motion information is fused to generate the gaze prediction maps of the dynamic video. Compared with the state-of-art models on DHF1K dataset, the CC, AUC-j, sAUC, NSS are separately increased by 5.1%, 0.6%, 38.2% and 0.5%. Graphical abstract: Highlights: This paper proposes a gaze prediction algorithm based on LSTM convolution based on video frame correlation features (LSTM-CVFAF). In SGP-Net, the convolution Gaussian prior layer is used to simulate the bias phenomenon in eye fixation.And a TGP-Net composed of multiple ConvLSTM layers is proposed to learn temporal motion feature information between video frames. Fu-Net fuses the position and motion information extracted from SGP-Net and TGP-Net in a self-learning way, so as to obtain a more accurate video gaze prediction map.