Assessing future landscapes using enhanced mixed reality with semantic segmentation by deep learning. (April 2021)

• Record Type: Journal Article
• Authors: Kido, Daiki; Fukuda, Tomohiro; Yabuki, Nobuyoshi
• Is Part Of: Advanced engineering informatics. Volume 48(2021)
• Publication Date: 2021-04
• Subjects: Mixed reality -- Dynamic occlusion handling -- Landscape index estimation -- Landscape design -- Video communication -- Deep learning; GPU graphics processing unit -- GVI Green View Index -- IoU intersection over union -- MR mixed reality -- VR virtual reality; Computer-aided engineering -- Periodicals; Engineering -- Data processing -- Periodicals; 620.00285
• DOI: 10.1016/j.aei.2021.101281
• Languages: English
• ISSNs: 1474-0346

Abstract

Graphical abstract: Highlights: A mixed reality (MR) landscape visualization system with deep learning is proposed. Dynamic occlusion and Green View Index (GVI) estimation are implemented in MR. MR could visualize the appearance and GVI of current and designed landscapes. A game engine on a mobile device is combined with semantic segmentation online. Internet speed and latency were evaluated to ensure real-time MR rendering. Abstract: Architecture, engineering, and construction projects need to be promoted in harmony with the natural environment and with the aim of preserving people's living environment. At the planning and design stage, decision-makers and stakeholders share and assess landscape images during and after construction in order to avoid as much uncertainty as possible when performing environmental impact assessment. Given the lack of a standard visualization method for future landscapes that do not yet exist, mixed reality (MR), which overlays virtual content onto a real scene, has attracted attention in the field of landscape design. One challenge in MR is occlusion, which occurs when virtual objects obscure physical objects that should be rendered in the foreground. In MR-based landscape visualization, the distance between the MR camera and real objects located in front of the virtual objects might vary and might be large, causing difficulty for existing occlusion handling methods. In the process of landscape design, an evidence-based approach has also become important. Landscape index estimation using semantic segmentation by deep learning, which can recognize the surrounding environment, has been actively studied for landscape assessment. In this study, semantic segmentation by deep learning was integrated into an MR system to enable dynamic occlusion handling and landscape index estimation for both existing and designed landscape assessment. This system can be operated on a mobile device with video communication over the internet by connecting to real-time semantic segmentation on a high-performance personal computer. The applicability of the developed system is demonstrated through accuracy verification and case studies.