Multitask painting categorization by deep multibranch neural network. (30th November 2019)

• Record Type: Journal Article
• Authors: Bianco, Simone; Mazzini, Davide; Napoletano, Paolo; Schettini, Raimondo
• Is Part Of: Expert systems with applications. Volume 135(2019)
• Page Start: 90
• Page End: 101
• Publication Date: 2019-11-30
• Subjects: Painting categorization -- Painting style classification -- Painter recognition -- Deep convolutional neural network -- Multiresolution -- Multitask; 00-01 -- 99-00; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2019.05.036
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: A novel deep multibranch multitask neural network architecture. The different branches process the input image at different scales. A trainable crop strategy to feed branches with the most informative regions. The injection of hand-crafted features inside the network for painting categorization. A new dataset composed of 100k paintings from 1508 artists, 125 styles, 41 genres. Abstract: We propose a novel deep multibranch and multitask neural network for artist, style, and genre painting categorization. The multibranch approach allows us to exploit at the same time the coarse layout of the painting and the fine-grained structures by using painting crops at different resolutions that are wisely extracted using a Spatial Transformer Network trained to identify the most discriminative subregions of paintings. The effectiveness of the proposed network is proved in experiments that are performed on a new dataset originally sourced from wikiart.org and hosted by Kaggle, and made suitable for artist, style and genre multitask learning. The dataset here proposed and made available for research is named MultitaskPainting100k, and is composed by 100K paintings, 1508 artists, 125 styles and 41 genres annotated by human experts. Among the different variants of the proposed network, the best method achieves accuracy levels of 56.5%, 57.2%, and 63.6% on the MultitaskPainting100k dataset for the tasks of artist, style and genre prediction respectively.