Spatio-temporal union of subspaces for multi-body non-rigid structure-from-motion. (November 2017)

• Record Type: Journal Article
• Authors: Kumar, Suryansh; Dai, Yuchao; Li, Hongdong
• Is Part Of: Pattern recognition. Volume 71(2017:Nov.)
• Page Start: 428
• Page End: 443
• Publication Date: 2017-11
• Subjects: Structure from Motion (SfM) -- Subspace clustering -- Alternating Direction Method of Multipliers (ADMM) -- Deformable objects; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2017.05.014
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a joint segmentation and reconstruction framework to the challenging task of complex multi-body NRSfM by exploiting the inherent spatio-temporal union of subspace constraint. We propose to efficiently solve the resultant non-convex optimization problem based on the ADMM method. Extensive experimental results on both synthetic and real multi-body NRSfM datasets demonstrate the superior performance of our proposed framework. Abstract: Non-rigid structure-from-motion (NRSfM) has so far been mostly studied for recovering 3D structure of a single non-rigid/deforming object. To handle the real world challenging multiple deforming objects scenarios, existing methods either pre-segment different objects in the scene or treat multiple non-rigid objects as a whole to obtain the 3D non-rigid reconstruction. However, these methods fail to exploit the inherent structure in the problem as the solution of segmentation and the solution of reconstruction could not benefit each other. In this paper, we propose a unified framework to jointly segment and reconstruct multiple non-rigid objects. To compactly represent complex multi-body non-rigid scenes, we propose to exploit the structure of the scenes along both temporal direction and spatial direction, thus achieving a spatio-temporal representation. Specifically, we represent the 3D non-rigid deformations as lying in a union of subspaces along the temporal direction and represent the 3D trajectories as lying in the union of subspaces along the spatial direction. This spatio-temporal representation not only provides competitive 3D reconstruction but also outputs robust segmentation of multiple non-rigid objects. The resultant optimization problem is solved efficiently using the Alternating Direction Method of Multipliers (ADMM). Extensive experimental results on both synthetic and real multi-body NRSfM datasets demonstrate the superior performance of our proposed framework compared with the state-of-the-art methods.