On minimal representations of shallow ReLU networks. (April 2022)

• Record Type: Journal Article
• Authors: Dereich, Steffen; Kassing, Sebastian
• Is Part Of: Neural networks. Volume 148(2022)
• Page Start: 121
• Page End: 128
• Publication Date: 2022-04
• Subjects: Neural networks -- Shallow networks -- Minimal representations -- ReLU activation; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.01.006
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: The realization function of a shallow ReLU network is a continuous and piecewise affine function f : R d → R, where the domain R d is partitioned by a set of n hyperplanes into cells on which f is affine. We show that the minimal representation for f uses either n, n + 1 or n + 2 neurons and we characterize each of the three cases. In the particular case, where the input layer is one-dimensional, minimal representations always use at most n + 1 neurons but in all higher dimensional settings there are functions for which n + 2 neurons are needed. Then we show that the set of minimal networks representing f forms a C ∞ -submanifold M and we derive the dimension and the number of connected components of M . Additionally, we give a criterion for the hyperplanes that guarantees that a continuous, piecewise affine function is the realization function of an appropriate shallow ReLU network.