Learning to infer human attention in daily activities. (July 2020)

• Record Type: Journal Article
• Authors: Nan, Zhixiong; Shu, Tianmin; Gong, Ran; Wang, Shu; Wei, Ping; Zhu, Song-Chun; Zheng, Nanning
• Is Part Of: Pattern recognition. Volume 103(2020:Jul.)
• Publication Date: 2020-07
• Subjects: Human attention -- Deep neural network -- Attentional objects; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107314
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Different encoder-decoder architectures significantly affect the performance. The performance improves after considering the task encoding loss. The pretrain of the neural network contributes to better performance. Abstract: The first attention model in the computer science community is proposed in 1998. In the following years, human attention has been intensively studied. However, these studies mainly refer human attention as the image regions that draw the attention of a human (outside the image) who is looking at the image. In this paper, we infer the attention of a human inside a third-person view video where the human is doing a task, and define human attention as attentional objects that coincide with the task the human is doing. To infer human attention, we propose a deep neural network model that fuses both low-level human pose cue and high-level task encoding cue. Due to the lack of appropriate public datasets for studying this problem, we newly collect a video dataset in complex Virtual-Reality (VR) scenes. In the experiments, we widely compare our method with three other methods on this VR dataset. In addition, we re-annotate a public real dataset and conduct the extensional experiments on this real dataset. The experiment results validate the effectiveness of our method.