Correlation-based structural dropout for convolutional neural networks. (December 2021)

• Record Type: Journal Article
• Authors: Zeng, Yuyuan; Dai, Tao; Chen, Bin; Xia, Shu-Tao; Lu, Jian
• Is Part Of: Pattern recognition. Volume 120(2021)
• Publication Date: 2021-12
• Subjects: Over-fitting -- Regularization -- Dropout -- Convolutional neural networks; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108117
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: CorrDrop regularizes CNNs by dropping feature units based on feature correlation. A structural dropout method can effectively drop features in CNNs. Spatial-wise and channel-wise CorrDrop are proposed. Extensive experiments show the superiority of CorrDrop over other counterparts. Abstract: Convolutional neural networks (CNNs) easily suffer from the over-fitting problem since they are often over-parameterized in the case of small training datasets. The conventional dropout that drops feature units randomly works well for fully connected networks, but fails to regularize CNNs well due to high spatial correlation of the intermediate features, which allows the dropped information to flow through the network, thus leading to the problem of under-dropping. To better regularize CNNs, some structural dropout methods such as SpatialDropout and DropBlock have been proposed by dropping feature units in continuous regions randomly. However, these methods may suffer from the over-dropping problem by discarding the critical discriminative features, thus limiting the performance of CNNs. To address these issues, we propose a novel structural dropout method, Correlation based Dropout (CorrDrop), to regularize CNNs by dropping feature units based on feature correlation. Unlike the previous dropout methods, our CorrDrop can focus on the discriminative information and drops features in a spatial-wise or channel-wise manner. Extensive experiments on different datasets, network architectures, and various tasks (e.g., image classification and object localization) demonstrate the superiority of our method over other methods.