A Novel Low-Bit Quantization Strategy for Compressing Deep Neural Networks. (18th February 2020)

• Record Type: Journal Article
• Authors: Long, Xin; Zeng, XiangRong; Ben, Zongcheng; Zhou, Dianle; Zhang, Maojun
• Other Names: Franco Leonardo Academic Editor.
• Is Part Of: Computational intelligence and neuroscience. Volume 2020(2020)
• Publication Date: 2020-02-18
• Subjects: Neurosciences -- Data processing -- Periodicals; Computational intelligence -- Periodicals; Computational neuroscience -- Periodicals; 612.80285
• DOI: 10.1155/2020/7839064
• Languages: English
• ISSNs: 1687-5265

Abstract

Abstract : The increase in sophistication of neural network models in recent years has exponentially expanded memory consumption and computational cost, thereby hindering their applications on ASIC, FPGA, and other mobile devices. Therefore, compressing and accelerating the neural networks are necessary. In this study, we introduce a novel strategy to train low-bit networks with weights and activations quantized by several bits and address two corresponding fundamental issues. One is to approximate activations through low-bit discretization for decreasing network computational cost and dot-product memory. The other is to specify weight quantization and update mechanism for discrete weights to avoid gradient mismatch. With quantized low-bit weights and activations, the costly full-precision operation will be replaced by shift operation. We evaluate the proposed method on common datasets, and results show that this method can dramatically compress the neural network with slight accuracy loss.