A grid anchor based cropping approach exploiting image aesthetics, geometric composition, and semantics. (30th December 2021)

• Record Type: Journal Article
• Authors: Celona, Luigi; Ciocca, Gianluigi; Napoletano, Paolo
• Is Part Of: Expert systems with applications. Volume 186(2021)
• Publication Date: 2021-12-30
• Subjects: Image cropping -- Image aesthetics -- Image composition -- Semantic content -- Deep learning; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2021.115852
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Image cropping aims at the selection of the relevant part of an image maximizing its aesthetic quality and composition. The part of the image that needs to be removed is highly dependent on user preferences and can be related to image aesthetics, composition, informativeness, or other criteria. Since the concept of the perfect crop does not exist, but there are several cropping possibilities, recent cropping algorithms are trained to rank a set of crop candidates based on their compositional quality. To this end, several benchmark databases have been released that provide for each image a series of human-annotated crop candidates with corresponding scores. Many of the image cropping methods rely on a single criterion to define the best crop or crops in an image. However, a single criterion misses the complexity of human opinions which can differ in personal preferences and backgrounds. Motivated by this, we formulate the cropping problem as a ranking problem of candidate crop regions using a grid anchor based approach and multiple criteria. To evaluate the goodness of a crop region, we design a cropping method by combining three efficient and lightweight neural networks specifically designed to evaluate the quality of a crop in terms of aesthetics, composition, and semantics. Our results on standard datasets show that using more criteria yields better crops than state-of-the-art approaches. This result is also confirmed by a subjective study on user preferences that involved a panel of users. Highlights: A method evaluating candidate cropping regions using different quality criteria. A candidate region is evaluated in terms of aesthetic, composition, and semantics. The proposed method exploits efficient and lightweight convolutional neural networks. Experimental results confirm the goodness of the proposed approach.