Stateful human-centered visual captioning system to aid video surveillance. (September 2019)

• Record Type: Journal Article
• Authors: Saleem, Summra; Dilawari, Aniqa; Khan, Usman Ghani; Iqbal, Razi; Wan, Shaohua; Umer, Tariq
• Is Part Of: Computers & electrical engineering. Volume 78(2019)
• Page Start: 108
• Page End: 119
• Publication Date: 2019-09
• Subjects: Multi-line video captioning -- Attention -- Bi-directional LSTM -- Word embeddings -- Facial and visual features -- Hidden units; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2019.07.009
• Languages: English
• ISSNs: 0045-7906

Abstract

Highlights: This paper describes approaches of caption generation that help to integrate a form of attention with variants. There are two types of variants: a soft attention technique and a hard attention technique. Due to recent advancements in caption generation and employment of attention in machine translation, we further investigate those models, which involve salient part of a video and will generate its caption. By visualizing what and where the attention should be focused, we interpret the results of this framework. We also validate the advantages of attention in caption generation by using state of art performance on benchmark dataset. Abstract: The study of Natural Language Generation (NLG), especially how human beings narrate the world, assists in understanding of the visual world for surveillance. Our research proposes an effective technique to axiomatically develop multi-line textual description of visual data by exploiting deep Convolution Neural Networks (CNN). Textual description of visual data aids in providing textual tags for visual information. A human can retrieve elected videos from a repository based on visual tags. Videos contain more complex and detailed information than images and provide more language data. The proposed feats-rich model encodes the visual contents to visual and facial features using CNN architecture. Encoded features are passed to two layer LSTM units with attention mechanism, reducing the number of parameters by encompassing relevant details. Experimental results on Trecvid 2016 and UET-Surveillance dataset depict that model outperforms state-of-the-art methods by scoring BLEU score of 0.35 and 0.52, respectively.