SNAP: Shaping neural architectures progressively via information density criterion. (August 2021)

• Record Type: Journal Article
• Authors: Chen, Zhiqiang; Xu, Ting-Bing; Liao, Weijian; Li, Zhengcheng; Li, Jinpeng; Liu, Cheng-Lin; He, Huiguang
• Is Part Of: Pattern recognition. Volume 116(2021)
• Publication Date: 2021-08
• Subjects: Auto-generated neural architectures -- Information density -- Greedy strategy -- Progressively -- Efficient and adaptive; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.107923
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a progressive method SNAP to shape a given neural architecture to a more reasonable one progressively, which is inspired by the streamline of water droplet driven by the air resistance progressively. The proposed method is efficient due to the greedy strategy. And we give a detailed proof that the greedy strategy is reasonable in theory. We propose the information density criterion to induce the progressive pro- cess, which is both task-aware and device-aware. The experimental results show that the proposed method can significant- ly improve the performance of the given architecture. And it can achieve comparable or even better performance compared with the search based ar- chitecture auto-generated methods in no need of tremendous computation resources. Abstract: Excellent neural network architecture is built on the specific target task and device. As the target task or device is different, the neural architecture we need will be different, too. Rather than redesigning or searching a brand new one, adjusting the existing architecture automatically is an alternative yet efficient way. To this end, we propose a method to Shape the existing Neural Architectures Progressively (SNAP) to adapt the target task and device better. Inspired by the streamline of water drop shaped by air resistance, we define an information density criterion (play the role of resistance) to drive the network architecture reducing the size of the part with the lowest information density. Iteratively, a more adaptive architecture will be obtained progressively in a greedy way. Theoretically, we prove that the greedy strategy is reasonable and can shape a better architecture. Because of the small adjustment of architecture each time, new architecture can inherit the parameters in old architecture to avoid retraining it from scratch. So the proposed method is very efficient in no need of high computation cost. Experimental results show that proposed method can effectively improve the given network by adjusting its architecture. And it can generate different architectures for different tasks and devices to adapt them well. Compared with search-based auto-generated neural architectures, our approach can achieve comparable or even better performance in no need of tremendous computation resources.