A non-linear mapping representing human action recognition under missing modality problem in video data. (December 2021)

• Record Type: Journal Article
• Authors: Gharahdaghi, Aidin; Razzazi, Farbod; Amini, Arash
• Is Part Of: Measurement. Volume 186(2021)
• Publication Date: 2021-12
• Subjects: Missing modality -- Human action recognition -- RGB-D data; Weights and measures -- Periodicals; Measurement -- Periodicals; Measurement; Weights and measures; Periodicals; 530.8
• DOI: 10.1016/j.measurement.2021.110123
• Languages: English
• ISSNs: 0263-2241

Abstract

Abstract: Human action recognition by using standard video files is a well-studied problem in the literature. In this study, we assume to have access to single modality standard data of some actions (training data). Based on this data, we aim at identifying the actions that are present in a target modality video data without any explicit source–target relationship information. In this case, the training and test phases of the recognition task are based on different imaging modalities. Our goal in this paper is to introduce a mapping (a nonlinear operator) on both modalities such that the outcome shares some common features. These common features were then used to recognize the actions in each domain. Simulation results on MSRDailyActivity3D, MSRActionPairs, UTKinect-Action3D, and SBU Kinect interaction datasets showed that the introduced method outperforms state-of-the art methods with a success rate margin of 15% on average. Highlights: We proposed a nonlinear mapping in cross modal human action recognition. We proposed a cropping strategy to extract the salient segment of video frames. The method is developed without using any joint RGB-D auxiliary dataset.