Unsupervised virtual view synthesis from monocular video with generative adversarial warping. (December 2021)

• Record Type: Journal Article
• Authors: Wang, Xiaochuan; Liu, Yi; Li, Haisheng
• Is Part Of: Computers & electrical engineering. Volume 96:Part B(2021)
• Publication Date: 2021-12
• Subjects: View synthesis -- Unsupervised learning -- Depth-image-based rendering -- Generative adversarial network -- Perceptual constraints; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2021.107460
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Virtual view synthesis from monocular video is challenging, as it aims to infer photorealistic views from single reference view. Previous work have achieved acceptable visual quality, however, are heavily relied on supervision information, such as depth or pristine virtual view, which are inadequate in practical application. In this paper, an unsupervised virtual view synthesis method is proposed to get rid of the supervision information. Firstly, it embed a spatiotemporal generative adversarial network into traditional depth-image-based rendering framework with no explicit depth information provided. Secondly, it utilized novel perceptual constraints without relying on pristine images, including the blind synthesized image quality metric and no-reference structure similarity. The entire framework is fully convolutional, producing hallucinated results in an end-to-end way. Particularly, the whole framework is independent of supervision information. Experimental results demonstrate that the proposed method produces pleasant virtual views in comparison with supervised methods, thereby can be beneficial to practical applications. Graphical abstract: Highlights: An unsupervised virtual view synthesis approach from monocular video. Embedding traditional DIBR into generative network. Introducing perception constraints to optimize the visual quality. Presenting plausible synthesized results in both objective and subjective studies.