Automated segmentation of RGB-D images into a comprehensive set of building components using deep learning. (August 2020)

• Record Type: Journal Article
• Authors: Czerniawski, Thomas; Leite, Fernanda
• Is Part Of: Advanced engineering informatics. Volume 45(2020)
• Publication Date: 2020-08
• Subjects: Building information modeling -- Semantic segmentation -- Deep learning -- Class balancing -- RGB-D -- 3DFacilities; Computer-aided engineering -- Periodicals; Engineering -- Data processing -- Periodicals; 620.00285
• DOI: 10.1016/j.aei.2020.101131
• Languages: English
• ISSNs: 1474-0346

Abstract

Highlights: RGB-D semantic segmentation: 13 building component classes with an average IoU 0.50. Effective class balancing of a small semantic segmentation datasets. Inputting color, depth, and accelerometer data despite pre-training with only color. Abstract: Building information modeling (BIM) has a semantic scope that encompasses all building systems, e.g. architectural, structural, mechanical, electrical, and plumbing. Automated, comprehensive digital modeling of buildings will require methods for semantic segmentation of images and 3D reconstructions capable of recognizing all building component classes. However, prior building component recognition methods have had limited semantic coverage and are not easily combined or scaled. Here we show that a deep neural network can semantically segment RGB-D (i.e. color and depth) images into 13 building component classes simultaneously despite the use of a small training dataset with only 1490 object instances. For this task, the method achieves an average intersection over union (IoU) of 0.5. The dataset was designed using a common building taxonomy to ensure comprehensive semantic coverage and was collected from a diversity of buildings to ensure intra-class diversity. As a consequence of its semantic scope, it was necessary to perform pre-segmentation and 3D to 2D projection as leverage for dataset annotation. In creating our deep learning pipeline, we found that transfer learning, class balancing, and prevention of overfitting effectively overcame the dataset's borderline adequate class representation. Our results demonstrate how the semantic coverage of a building component recognition method can be scaled to include a larger diversity of building systems. We anticipate our method to be a starting point for broadening the scope of the semantic segmentation methods involved in digital modeling of buildings.