LOss-Based SensiTivity rEgulaRization: Towards deep sparse neural networks. (February 2022)

• Record Type: Journal Article
• Authors: Tartaglione, Enzo; Bragagnolo, Andrea; Fiandrotti, Attilio; Grangetto, Marco
• Is Part Of: Neural networks. Volume 146(2022)
• Page Start: 230
• Page End: 237
• Publication Date: 2022-02
• Subjects: Pruning -- Regularization -- Deep learning -- Sparsity; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2021.11.029
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: LOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.