Self-supervised learning of scene flow with occlusion handling through feature masking. (July 2023)

• Record Type: Journal Article
• Authors: Xiang, Xuezhi; Abdein, Rokia; Lv, Ning; Yang, Jie
• Is Part Of: Pattern recognition. Volume 139(2023)
• Publication Date: 2023-07
• Subjects: Optical flow -- Depth estimation -- Camera pose -- Rigidity segmentation -- Occlusion handling -- Deformable decoder; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109487
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a feature masking method to handle the occlusion in optical flow and depth estimation. We propose a deformable decoder to improve the structural details in occluded and non-occluded regions of optical flow and depth. We propose an Update-flow block to refine the optical flow. Abstract: In this work, we improve the optical flow and depth based on the important observation that they should share the geometric structure of the reference image. We initially propose a feature masking method to reduce the occlusion impact on the optical flow and depth by producing preliminary motion features that share the structure of the reference image. In addition, we propose a deformable decoder that learns the geometrical structure of the reference image in the form of a group of offsets and uses them to adapt the motion features of flow and depth maps, thereby preventing incorrect propagation in occluded regions and providing more structural details in the other regions. Furthermore, we recursively update the optical flow with self-supervised cues learned from the rigid flow and optical flow. Our method achieves a new state-of-the-art result for the optical flow on the KITTI 2015 benchmark with F1 = 11.17%.