A coarse-to-fine approach for dynamic-to-static image translation. (March 2022)

• Record Type: Journal Article
• Authors: Wang, Teng; Wu, Lin; Sun, Changyin
• Is Part Of: Pattern recognition. Volume 123(2022)
• Publication Date: 2022-03
• Subjects: Dynamic-to-static image translation -- Shadow detection -- Attention mechanism -- Visual place recognition; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108373
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Dynamic-to-static image translation is delicately formulated as an image inpainting-like problem, and a novel coarse-to-fine framework is proposed. A simple but effective strategy is designed to handle the presences of object shadows. A mutual texture-structure attention module is proposed to enhance the recovery of textures and structures in dynamic areas. We generate a new test dataset with high diversity to supplement the existing test dataset. Extensive experimental results demonstrate the superiority of our proposed approach. Abstract: Dynamic-to-static image translation aims to convert the dynamic scene into static so that dynamic elements are eliminated from the image. Recent works typically see the problem as an image-to-image translation task, and perform the learned feature mapping over the whole dynamic image to synthesize the static image, which leads to unnecessary detail loss in original static regions. To that end, we delicately formulate it as an image inpainting-like problem to fill the missing static pixels in dynamic regions while retaining original static regions. We achieve this by proposing a coarse-to-fine framework. At coarse stage, we utilize a simple encoder-decoder network to rough out the static image. Using the coarse predicted image, we explicitly infer a more accurate dynamic mask to identify both dynamic objects and their shadows, so that the task could be effectively converted to an image inpainting problem. At fine stage, we recover the missing static pixels in the estimated dynamic regions on the basis of their coarse predictions. We enhance the coarse predicted contents by proposing a mutual texture-structure attention module, which enables the dynamic regions to borrow textures and structures separately from distant locations based on contextual similarity. Several losses are combined as the training objective function to generate excellent results with global consistency and fine details. Qualitative and quantitative experiments verify the superiority of our method in restoring high-quality static contents over state-of-the-art models. In addition, we evaluate the usefulness of the recovered static images by using them as query images to improve visual place recognition in dynamic scenes.