Versatile recurrent neural network for wide types of video restoration. (June 2023)

• Record Type: Journal Article
• Authors: Wang, Yadong; Bai, Xiangzhi
• Is Part Of: Pattern recognition. Volume 138(2023)
• Publication Date: 2023-06
• Subjects: RNN -- Video restoration -- Versatile -- Efficient; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109360
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A general video restoration paradigm is proposed via analyzing video degradation mechanism. Hidden states flow bidirectionally, making full use of the temporal-spatial information. Fusing useful features of neighboring frames to help reconstruct current frame. Fast processing speed and excellent performance on four types of video restoration tasks. Abstract: Video shooting of natural scenes often suffers from various serious degradation, such as motion blur, impact of atmospheric turbulence, random noise and resolution reduction, etc. Different from the maturity of image restoration research, video restoration is much more complicated so that it lacks effective general method. Here, we present a versatile recurrent neural network (VRNN) to handle wide types of video degradation and generate stable videos with ideal clarity. We complete the design of VRNN through deducing a general video restoration paradigm that reveals the importance of simultaneously utilizing past and future information for restoring current frame. Specifically, we propose a novel RNN cell in which hidden state flows in bidirections, enriching temporal information contained in the extracted features. Furthermore, a feature fusion module involves temporal and spatial attention processing is designed to refine features of neighbouring frames and help reconstruct current frame. Extensive experiments on well-known public datasets (including four different kinds of video restoration tasks, with a total of 35, 666 videos and 515, 774 frames) show that the proposed VRNN achieves 1–4 dB of PSNR increasing or several times less of computational complexity in all tasks against state-of-the-art methods, manifesting the versatile and efficient ability of proposed VRNN in wide types of video restoration.