Video summarization with a convolutional attentive adversarial network. (November 2022)

• Record Type: Journal Article
• Authors: Liang, Guoqiang; Lv, Yanbing; Li, Shucheng; Zhang, Shizhou; Zhang, Yanning
• Is Part Of: Pattern recognition. Volume 131(2022)
• Publication Date: 2022-11
• Subjects: Video summarization -- Generative adversarial network -- Self attention; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.108840
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We integrate the self-attention mechanism and a fully convolutional sequence network to capture the global and local temporal dependencies of video frames. A convolutional attentive generative adversarial network is designed for unsupervised video summarization. By discarding recurrent structures, our generator can be paralleled much easier. Experiment results show that our method not only achieves better or comparable performance within unsupervised methods, but also is superior to most of the published supervised approaches. Abstract: With the explosive growth of video data, video summarization, which attempts to seek the minimum subset of frames while still conveying the main story, has become one of the hottest topics. Nowadays, substantial achievements have been made by supervised learning techniques, especially after the emergence of deep learning. However, it is extremely expensive and difficult to construct a large-scale video summarization dataset through human annotation. To address this problem, we propose a convolutional attentive adversarial network (CAAN), whose key idea is to build a deep summarizer in an unsupervised way. Upon the generative adversarial network, our overall framework consists of a generator and a discriminator. The former predicts importance scores for all the frames of a video while the latter tries to distinguish the score-weighted frame features from original frame features. To capture the global and local temporal relationship of video frames, the generator employs a fully convolutional sequence network to build global representation of a video, and an attention-based network to predict normalized importance scores. To optimize the parameters, our objective function is composed of three loss functions, which can guide the frame-level importance score prediction collaboratively. To validate this proposed method, we have conducted extensive experiments on two public benchmarks SumMe and TVSum. The results show the superiority of our proposed method against other state-of-the-art unsupervised approaches. Our method even outperforms some published supervised approaches.