Acoustic model training based on node-wise weight boundary model for fast and small-footprint deep neural networks. (November 2017)

• Record Type: Journal Article
• Authors: Takeda, Ryu; Nakadai, Kazuhiro; Komatani, Kazunori
• Is Part Of: Computer speech & language. Volume 46(2017)
• Page Start: 461
• Page End: 480
• Publication Date: 2017-11
• Subjects: Deep neural network -- Acoustic model -- Small-footprint -- Quantization -- Node-pruning; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2017.02.002
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: A node-wise boundary model for neural networks-based acoustic model is proposed. A fast implementation scheme using a look-up table on CPUs is also proposed. 40% increase in speed of the neural network's forward calculation is achieved. 2-bit neural networks maintained word accuracy equivalent to that of the 32-bit ones. Abstract: Our goal for this study is to enable the development of discrete deep neural networks (NNs), some parameters of which are discretized, as small-footprint and fast NNs for acoustic models. Three essential requirements should be met for achieving this goal; 1) the reduction in discretization errors, 2) implementation for fast processing and 3) node-size reduction of DNNs. We propose a weight-parameter model and its training algorithm for 1), an implementation scheme using a look-up table on general-purpose CPUs for 2), and a layer-biased node-pruning method for 3). The first proposed method can set proper boundaries of discretization at each NN node, resulting in reduction in discretization errors. The second method can reduce the memory usage of NNs within the cache size of the CPU by encoding the parameters of NNs. The last method can reduce the network size of the quantized DNNs by measuring the activity of each node at each layer and pruning them with a layer-dependent score. Experiments with 2-bit discrete NNs showed that our training algorithm maintained almost the same word accuracy as with 8-bit discrete NNs. We achieved a 95% reduction of memory usage and a 74% increase in speed of an NN's forward calculation.