Vision-based method for semantic information extraction in construction by integrating deep learning object detection and image captioning. (August 2022)

• Record Type: Journal Article
• Authors: Wang, Yiheng; Xiao, Bo; Bouferguene, Ahmed; Al-Hussein, Mohamed; Li, Heng
• Is Part Of: Advanced engineering informatics. Volume 53(2022)
• Publication Date: 2022-08
• Subjects: Deep learning -- Image captioning -- Semantic information extraction -- Object detection -- Construction engineering; Computer-aided engineering -- Periodicals; Engineering -- Data processing -- Periodicals; 620.00285
• DOI: 10.1016/j.aei.2022.101699
• Languages: English
• ISSNs: 1474-0346

Abstract

Graphical abstract: Highlights: Proposes a novel method for semantic information extraction from images in construction engineering. Integrates deep learning object detection and image captioning. Achieves the Consensus-based Image Description Evaluation (CIDEr) of 1.84 in experiments. Develops an algorithm for visualizing construction scene graph. Abstract: Recently, vision-based monitoring has been widely adopted in construction management to improve crew productivity, reduce safety risks, and facilitate site planning. However, automated retrieval of semantic information (e.g., objects, activities, and interactions between objects) from construction images remains challenging due to the complex nature of construction sites. This paper proposes a novel semantic information extraction method by integrating deep learning object detection and image captioning, which aims to explore salient information from construction images or videos. In the proposed method, object detection has been employed as an encoder to extract the feature maps of construction object zones and the holistic image. The image captioning has been selected as the decoder to extract the semantic information. A post-processing method has been proposed to parse the semantic information into a graph format for better accessibility and visualization. In experiments, the proposed method has achieved the Consensus-based Image Description Evaluation (CIDEr) of 1.84. By adopting the proposed method, semantic information behind construction images can be presented to construction managers to assist their decision-making.