Measuring the usefulness of hidden units in Boltzmann machines with mutual information. (April 2015)

• Record Type: Journal Article
• Authors: Berglund, Mathias; Raiko, Tapani; Cho, Kyunghyun
• Is Part Of: Neural networks. Volume 64(2015:Apr.)
• Page Start: 12
• Page End: 18
• Publication Date: 2015-04
• Subjects: Deep learning -- Restricted Boltzmann machine -- Deep Boltzmann machine -- Pruning -- Structural learning -- Mutual information; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2014.09.004
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure is fast to compute, and serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results. We confirm experimentally that the proposed measure indicates how much the performance of the model drops when some of the units of an RBM are pruned away. We demonstrate the usefulness of the measure for early detection of poor training in DBMs.