Video super-resolution via pre-frame constrained and deep-feature enhanced sparse reconstruction. (April 2020)

• Record Type: Journal Article
• Authors: Lai, Qiuxia; Nie, Yongwei; Sun, Hanqiu; Xu, Qiang; Zhang, Zhensong; Xiao, Mingyu
• Is Part Of: Pattern recognition. Volume 100(2020:Apr.)
• Publication Date: 2020-04
• Subjects: Video super resolution -- Sparse representation -- Deep features -- Temporal coherence; 00-01 -- 99-00; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2019.107139
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel sparse reconstruction formulation is introduced for video super-resolution, which uses the previous estimated high-resolution frame as regularization to guarantee the temporal coherence. Deep features are incorporated into the sparse reconstruction framework to enhance the dictionary which benefit the whole super-resolution process. An effective dictionary updating strategy is proposed that updates the dictionaries regularly utilizing the newly reconstructed frames. A joint bilateral filter is utilized to remove reconstruction noises and transfer details. Experiments show the effectiveness of the proposed method compared with previous approaches. Abstract: This paper presents a new video super-resolution (SR) method that can generate high-quality and temporally coherent high-resolution (HR) videos. Starting from the traditional sparse reconstruction framework that works well for image SR, we improve it significantly from the following aspects to obtain an effect video SR method. Firstly, to enhance the temporal coherence between adjacent HR frames, once a HR frame is estimated, we use it to guide the sparse reconstruction of the next low-resolution frame. Secondly, instead of using just hand-craft features, we further incorporate deep features generated by VGG16 into our sparse reconstruction based video SR method. Thirdly, we constantly update the dictionary, which is the core of the sparse reconstruction, by making use of the previously estimated HR frame. Finally, after the HR video is reconstructed, we use a joint bilateral filter to post-process it to remove artifacts and transfer image details. Experiments demonstrate that the proposed four strategies effectively improve our final results. In most of the experiments of this paper, our results are better than those produced by the latest deep learning based approaches.