Unsupervised learning of optical flow with patch consistency and occlusion estimation. (July 2020)

• Record Type: Journal Article
• Authors: Ren, Zhe; Yan, Junchi; Yang, Xiaokang; Yuille, Alan; Zha, Hongyuan
• Is Part Of: Pattern recognition. Volume 103(2020:Jul.)
• Publication Date: 2020-07
• Subjects: Patch consistency -- Optical flow estimation -- Occlusion estimation -- Unsupervised learning -- Deep learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2019.107191
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A patch-based census constancy loss is proposed by patch-based warping to further improve the performance of unsupervised optical flow estimation. A parallel decoder branch is devised for occlusion mask learning in an unsupervised way. The pseudo label generated from the forward-backward consistency check is used to derive a mask loss for occlusion mask learning. Our method achieves the-state-of-the-art results among the unsupervised learning methods that are using the FlowNet-liked network structure. Abstract: Recent works have shown that deep networks can be trained for optical flow estimation without supervision. Based on the photometric constancy assumption, most of these methods adopt the reconstruction loss as the supervision by point-based backward warping. Inspired by the traditional patch matching based approaches, we propose a patch-based consistency to improve the vanilla unsupervised learning method Ren et al. [1]. Instead of only comparing the corresponding pixel intensity, we locate the correspondence by using the image patches with census transform, which is more robust for the illumination variation and occlusion. Moreover, a novel parallel branch is devised to estimate a soft occlusion mask jointly in an unsupervised way. The mask is adopted to weight our patch-based consistency loss to alleviate the influence of the occlusion. The plenty of experiments have been implemented on Flying Chairs, KITTI and MPI-Sintel benchmarks. The results show that our method is efficient and outperforms the peer unsupervised learning methods that are using the FlowNet-liked network.