Virtual restoration of the colored paintings on weathered beams in the Forbidden City using multiple deep learning algorithms. (October 2021)

• Record Type: Journal Article
• Authors: Zou, Zheng; Zhao, Peng; Zhao, Xuefeng
• Is Part Of: Advanced engineering informatics. Volume 50(2021)
• Publication Date: 2021-10
• Subjects: Deep learning -- Heritage buildings -- Pix2pix -- U-Net -- Virtual restoration -- GAN -- Semantic segmentation; Computer-aided engineering -- Periodicals; Engineering -- Data processing -- Periodicals; 620.00285
• DOI: 10.1016/j.aei.2021.101421
• Languages: English
• ISSNs: 1474-0346

Abstract

Abstract: The colored paintings on the surfaces of ancient Chinese buildings have suffered from the wind and the sun and produced many defects such as paint loss, blurring, and color distortion. Previous methods have not been able to virtually repair them well. Therefore, this paper proposed a virtual restoration method for the weathered beams in the Forbidden City using multiple deep learning algorithms. Instead of using only one technology to restore paintings, this paper divided the painting into 3 parts, i.e., the background, the golden edges, and the dragon patterns, and restored them in different technology. For the background, this paper transformed the problem of unrecognizable color restoration into a semantic segmentation problem using U-Net MobileNet. For the golden edges, this paper used traditional image processing technology to obtain them from the color maps generated by the semantic segmentation algorithm. For the dragon patterns, after sketching the skeletons according to the dragon patterns, the image translation algorithm Pix2pix was applied to generate a realistic dragon pattern. Finally, the three repair results are superimposed to complete the repair. The virtually restored paintings can provide reference and guidance for traditional manual restoration and help the restorers to imagine what they looked like before oxidation, thus alleviating some repetitive work and reducing the complexity of restoration. Besides, each step of the method could form a layer, so that the restorers could overlay or modify them as they wish.