Weakly-supervised pre-training for 3D human pose estimation via perspective knowledge. (July 2023)

• Record Type: Journal Article
• Authors: Qiu, Zhongwei; Qiu, Kai; Fu, Jianlong; Fu, Dongmei
• Is Part Of: Pattern recognition. Volume 139(2023)
• Publication Date: 2023-07
• Subjects: Human pose estimation -- Pre-training -- Relative depth -- Weakly-supervised; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109497
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: To the best of our knowledge, we are the first one, who uses the perspective knowledge to generate relative depths of keypoints for the pretraining of 3D human pose estimation. It's less costly since no need of 3D pose annotation or artificial relative depth labeling. We propose a large-scale dataset (MCPC), with over 53k images and 220k instances, with the annotation of relative depth. These data could help relieve the need for large amounts of 3D pose annotations and improve the generalization ability of the 3D pose model. We propose the WSP approach and pre-train it on MCPC dataset. After finetuning WSP on 3D datasets, our approach outperforms previous approaches and achieves state-of-the-art results on two 3D pose estimation benchmarks. Abstract: Modern deep learning-based 3D pose estimation approaches require plenty of 3D pose annotations. However, existing 3D datasets lack diversity, which limits the performance of current methods and their generalization ability. Although existing methods utilize 2D pose annotations to help 3D pose estimation, they mainly focus on extracting 2D structural constraints from 2D poses, ignoring the 3D information hidden in the images. In this paper, we propose a novel method to extract weak 3D information directly from 2D images without 3D pose supervision. Firstly, we utilize 2D pose annotations and perspective prior knowledge to generate the relative depth of human joints. Then, we collect a 2D pose dataset (MCPC) and generate relative depth labels. Based on MCPC, we propose a weakly-supervised pre-training (WSP) strategy to distinguish the depth relationship between two points in an image. WSP enables the learning of the relative depth of two keypoints on lots of in-the-wild images, which is more capable of predicting depth and generalization ability for 3D human pose estimation. After fine-tuning the pose model on 3D pose datasets, WSP achieves state-of-the-art results on two widely-used benchmarks.