New optimization algorithms for neural network training using operator splitting techniques. (June 2020)

• Record Type: Journal Article
• Authors: Alecsa, Cristian Daniel; Pinţa, Titus; Boros, Imre
• Is Part Of: Neural networks. Volume 126(2020)
• Page Start: 178
• Page End: 190
• Publication Date: 2020-06
• Subjects: Neural network -- MNIST -- CIFAR10 -- Splitting -- Nesterov -- Dynamical system; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2020.03.018
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: In the following paper we present a new type of optimization algorithms adapted for neural network training. These algorithms are based upon sequential operator splitting technique for some associated dynamical systems. Furthermore, we investigate through numerical simulations the empirical rate of convergence of these iterative schemes toward a local minimum of the loss function, with some suitable choices of the underlying hyper-parameters. We validate the convergence of these optimizers using the results of the accuracy and of the loss function on the MNIST, MNIST-Fashion and CIFAR 10 classification datasets.