Mitigating severe over-parameterization in deep convolutional neural networks through forced feature abstraction and compression with an entropy-based heuristic. (November 2021)

• Record Type: Journal Article
• Authors: Gowdra, Nidhi; Sinha, Roopak; MacDonell, Stephen; Yan, Wei Qi
• Is Part Of: Pattern recognition. Volume 119(2021)
• Publication Date: 2021-11
• Subjects: Convolutional neural networks (CNNs) -- Depth redundancy -- Entropy -- Feature compression -- EBCLE; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108057
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose an accurate heuristic to determine an upper limit to convolutional depth using Shannon's entropy measure for forced feature map compression and abstraction which reduces training time by 24.99%-78.59% across various CNN architectures and datasets. Our experiments empirically validate and support the findings presented in [9] and [20] that, deep CNN models behave as a collection of ensemble networks and shallower CNN models can learn the same functional representations as deeper models with a reduction in the associated trade-off of relative model training time. Abstract: Convolutional Neural Networks (CNNs) such as ResNet-50, DenseNet-40 and ResNeXt-56 are severely over-parameterized, necessitating a consequent increase in the computational resources required for model training which scales exponentially for increments in model depth. In this paper, we propose an Entropy-Based Convolutional Layer Estimation (EBCLE) heuristic which is robust and simple, yet effective in resolving the problem of over-parameterization with regards to network depth of CNN model. The EBCLE heuristic employs a priori knowledge of the entropic data distribution of input datasets to determine an upper bound for convolutional network depth, beyond which identity transformations are prevalent offering insignificant contributions for enhancing model performance. Restricting depth redundancies by forcing feature compression and abstraction restricts over-parameterization while decreasing training time by 24.99% - 78.59% without degradation in model performance. We present empirical evidence to emphasize the relative effectiveness of broader, yet shallower models trained using the EBCLE heuristic, which maintains or outperforms baseline classification accuracies of narrower yet deeper models. The EBCLE heuristic is architecturally agnostic and EBCLE based CNN models restrict depth redundancies resulting in enhanced utilization of the available computational resources. The proposed EBCLE heuristic is a compelling technique for researchers to analytically justify their HyperParameter (HP) choices for CNNs. Empirical validation of the EBCLE heuristic in training CNN models was established on five benchmarking datasets (ImageNet32, CIFAR-10/100, STL-10, MNIST) and four network architectures (DenseNet, ResNet, ResNeXt and EfficientNet B0-B2) with appropriate statistical tests employed to infer any conclusive claims presented in this paper.