A comparison of deep networks with ReLU activation function and linear spline-type methods. (February 2019)

• Record Type: Journal Article
• Authors: Eckle, Konstantin; Schmidt-Hieber, Johannes
• Is Part Of: Neural networks. Volume 110(2019)
• Page Start: 232
• Page End: 242
• Publication Date: 2019-02
• Subjects: Deep neural networks -- Nonparametric regression -- Splines -- MARS -- Faber–Schauder system -- Rates of convergence; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2018.11.005
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Deep neural networks (DNNs) generate much richer function spaces than shallow networks. Since the function spaces induced by shallow networks have several approximation theoretic drawbacks, this explains, however, not necessarily the success of deep networks. In this article we take another route by comparing the expressive power of DNNs with ReLU activation function to linear spline methods. We show that MARS (multivariate adaptive regression splines) is improper learnable by DNNs in the sense that for any given function that can be expressed as a function in MARS with M parameters there exists a multilayer neural network with O ( M log ( M ∕ ε ) ) parameters that approximates this function up to sup-norm error ε . We show a similar result for expansions with respect to the Faber–Schauder system. Based on this, we derive risk comparison inequalities that bound the statistical risk of fitting a neural network by the statistical risk of spline-based methods. This shows that deep networks perform better or only slightly worse than the considered spline methods. We provide a constructive proof for the function approximations.