Diluted binary neural network. (August 2023)

• Record Type: Journal Article
• Authors: Lin, Yuhan; Niu, Lingfeng; Xiao, Yang; Zhou, Ruizhi
• Is Part Of: Pattern recognition. Volume 140(2023)
• Publication Date: 2023-08
• Subjects: Model compression -- Network quantization -- Binary neural network -- Ternary neural network -- Sparse regularization; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109556
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We provide a new insight on the reason of BNNs' performance degradation. A novel DBNN model is proposed to handle the accuracy drop of BNNs. A sparsity-binarization scheme for weight is given to avoid mandatory representation. A stable binarization strategy for activation is developed with layer normalization. A customized proximal gradient method is designed to derive diluted binary weights. Abstract: Binary neural networks (BNNs) are promising on resource-constrained devices because they reduce memory consumption and accelerate inference effectively. However, they are still potential on performance improvement. Prior studies attribute performance degradation of BNNs to limited representation ability and gradient mismatch. In this paper, we find that it also results from the mandatory representation of small full-precision auxiliary weights to large values. To tackle with this issue, we propose an approach dubbed as Diluted Binary Neural Network (DBNN). Besides avoiding mandatory representation effectively, the proposed DBNN also alleviates sign flip problem to a large extent. For activations, we jointly minimize quantization error and maximize information entropy to develop the binarization scheme. Compared with existing sparsity-binarization approaches, DBNN trains network from scratch without other procedures and achieves larger sparsity. Experiments on several datasets with various networks demonstrate the superiority of our approach.