SelfVIO: Self-supervised deep monocular Visual–Inertial Odometry and depth estimation. (June 2022)

• Record Type: Journal Article
• Authors: Almalioglu, Yasin; Turan, Mehmet; Saputra, Muhamad Risqi U.; de Gusmão, Pedro P.B.; Markham, Andrew; Trigoni, Niki
• Is Part Of: Neural networks. Volume 150(2022)
• Page Start: 119
• Page End: 136
• Publication Date: 2022-06
• Subjects: Self-supervised learning -- Geometry reconstruction -- Machine perception -- Generative adversarial networks -- Deep sensor fusion -- visual–inertial odometry; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.03.005
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: In the last decade, numerous supervised deep learning approaches have been proposed for visual–inertial odometry (VIO) and depth map estimation, which require large amounts of labelled data. To overcome the data limitation, self-supervised learning has emerged as a promising alternative that exploits constraints such as geometric and photometric consistency in the scene. In this study, we present a novel self-supervised deep learning-based VIO and depth map recovery approach (SelfVIO) using adversarial training and self-adaptive visual–inertial sensor fusion. SelfVIO learns the joint estimation of 6 degrees-of-freedom (6-DoF) ego-motion and a depth map of the scene from unlabelled monocular RGB image sequences and inertial measurement unit (IMU) readings. The proposed approach is able to perform VIO without requiring IMU intrinsic parameters and/or extrinsic calibration between IMU and the camera. We provide comprehensive quantitative and qualitative evaluations of the proposed framework and compare its performance with state-of-the-art VIO, VO, and visual simultaneous localization and mapping (VSLAM) approaches on the KITTI, EuRoC and Cityscapes datasets. Detailed comparisons prove that SelfVIO outperforms state-of-the-art VIO approaches in terms of pose estimation and depth recovery, making it a promising approach among existing methods in the literature.