A composition-oriented aesthetic view recommendation network supervised by the simplified golden ratio theory. (1st June 2022)

• Record Type: Journal Article
• Authors: Wang, Yaoting; Ke, Yongzhen; Wang, Kai; Guo, Jing; Qin, Fan
• Is Part Of: Expert systems with applications. Volume 195(2022)
• Publication Date: 2022-06-01
• Subjects: Aesthetic view recommendation -- Image aesthetic quality assessment -- Composition classification -- Image cropping -- Distillation model; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.116500
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Composition is the most vital element in photography. Previous research on automatic image cropping or view recommendation merely studied the aesthetic value of the cropping image without explicitly regarded the rules of composition. This work proposes a composition-oriented aesthetic view recommendation network supervised by the simplified golden ratio theory (CAVR-Net), compared to other methods that output only aesthetic scores, which outputs a sequence of local views with aesthetic rank and composition category for an image and helps to enhance the interpretation of aesthetic perception. We adopt a model distillation technique (teacher–student framework) to train the student model CAVR-S supervised by two teacher models including a composition prediction network (CPN) and an aesthetic evaluation network (AEN). In addition, for the challenging candidate crops selection problem in the image cropping task, we propose a candidate crops extraction scheme based on the simplified golden ratio theory, which reduces the training cost and improves the model performance. The CAVR-Net achieves state-of-the-art performance on two benchmark datasets in the image cropping task and a real-time recommendation efficiency of 80+ fps in the view recommendation task. Our method has essential application value in intelligent photography guidance system or intelligent image analysis system based on image composition, such as photography guide, automatic post-processing, automatic extraction of video frames. Highlights: A view recommendation model constrained by aesthetic evaluation and composition. A Simplified Golden Ratio Theory-based candidate box extraction. A dataset is constructed using the proposed candidate box selection strategy. An aesthetic evaluation network and a multitask loss function are proposed.