Capturing the spatio‐temporal continuity for video semantic segmentation. Issue 14 (1st December 2019)

• Record Type: Journal Article
• Authors: Chen, Xin; Wu, Aming; Han, Yahong
• Is Part Of: IET image processing. Volume 13:Issue 14(2019)
• Page Start: 2813
• Page End: 2820
• Publication Date: 2019-12-01
• Subjects: feature extraction -- image segmentation -- image representation -- video signal processing -- neural nets -- probability; video semantic segmentation -- image semantic segmentation -- convolutional neural network -- image segmentation algorithms -- video frame -- temporal region continuity inherent -- videos -- deep neural network architecture -- newly devised spatio‐temporal continuity -- encoding network -- STC module -- decoding network -- high‐level feature map -- STC feature map -- current feature representation -- consecutive video frames -- segmentation result; Image processing -- Periodicals; 621.36705
• DOI: 10.1049/iet-ipr.2018.6479
• Languages: English
• ISSNs: 1751-9659

Abstract

Abstract : In recent years, image semantic segmentation based on a convolutional neural network has achieved many advances. However, the development of video semantic segmentation is relatively slow. Directly applying the image segmentation algorithms to each video frame separately may ignore the temporal region continuity inherent in videos. In this study, the authors propose a novel deep neural network architecture with a newly devised spatio‐temporal continuity (STC) module for video semantic segmentation. Particularly, the architecture includes an encoding network, an STC module, and a decoding network. The encoding network is used to extract a high‐level feature map. The STC module then uses the high‐level feature map as input to extract the STC feature map. For decoding, they use four dilated convolutional layers to obtain more abstract representation and a deconvolutional layer to increase the size of the representation. Finally, they fuse the current feature representation and the previous feature representation and get the class probabilities. Thus, this architecture receives a sequence of consecutive video frames and outputs the segmentation result of the current frame. They extensively evaluate the proposed approach on the CamVid and KITTI datasets. Compared with other methods, the authors' approach not only achieves competitive performance but also has lower complexity.