Augmented saliency model using automatic 3D head pose detection and learned gaze following in natural scenes. (November 2015)

• Record Type: Journal Article
• Authors: Parks, Daniel; Borji, Ali; Itti, Laurent
• Is Part Of: Vision research. Volume 116:Part B(2015)
• Page Start: 113
• Page End: 126
• Publication Date: 2015-11
• Subjects: Visual attention -- Eye movements -- Gaze following -- Head pose detection -- Saliency modeling -- Fixation prediction; Vision -- Periodicals; 573.88
• DOI: 10.1016/j.visres.2014.10.027
• Languages: English
• ISSNs: 0042-6989

Abstract

Highlights: Learn gaze following + head fixation probability maps relative to actor's head pose. Weight cues based on type of current fixation (head/nonhead). Integrate head pose information with bottom-up saliency to make combined model. Integrate head pose detection to make combined model fully automatic. Abstract: Previous studies have shown that gaze direction of actors in a scene influences eye movements of passive observers during free-viewing (Castelhano, Wieth, & Henderson, 2007; Borji, Parks, & Itti, 2014). However, no computational model has been proposed to combine bottom-up saliency with actor's head pose and gaze direction for predicting where observers look. Here, we first learn probability maps that predict fixations leaving head regions (gaze following fixations), as well as fixations on head regions (head fixations), both dependent on the actor's head size and pose angle. We then learn a combination of gaze following, head region, and bottom-up saliency maps with a Markov chain composed of head region and non-head region states. This simple structure allows us to inspect the model and make comments about the nature of eye movements originating from heads as opposed to other regions. Here, we assume perfect knowledge of actor head pose direction (from an oracle). The combined model, which we call the Dynamic Weighting of Cues model (DWOC), explains observers' fixations significantly better than each of the constituent components. Finally, in a fully automatic combined model, we replace the oracle head pose direction data with detections from a computer vision model of head pose. Using these (imperfect) automated detections, we again find that the combined model significantly outperforms its individual components. Our work extends the engineering and scientific applications of saliency models and helps better understand mechanisms of visual attention.