A theory of capacity and sparse neural encoding. (November 2021)

• Record Type: Journal Article
• Authors: Baldi, Pierre; Vershynin, Roman
• Is Part Of: Neural networks. Volume 143(2021)
• Page Start: 12
• Page End: 27
• Publication Date: 2021-11
• Subjects: Neural capacity -- Sparse representations -- Sparse encoding -- Neural maps -- Threshold functions; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2021.05.005
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Motivated by biological considerations, we study sparse neural maps from an input layer to a target layer with sparse activity, and specifically the problem of storing K input-target associations ( x, y ), or memories, when the target vectors y are sparse. We mathematically prove that K undergoes a phase transition and that in general, and somewhat paradoxically, sparsity in the target layers increases the storage capacity of the map. The target vectors can be chosen arbitrarily, including in random fashion, and the memories can be both encoded and decoded by networks trained using local learning rules, including the simple Hebb rule. These results are robust under a variety of statistical assumptions on the data. The proofs rely on elegant properties of random polytopes and sub-gaussian random vector variables. Open problems and connections to capacity theories and polynomial threshold maps are discussed.