Neural network training as a dissipative process. (September 2016)

• Record Type: Journal Article
• Authors: Gori, Marco; Maggini, Marco; Rossi, Alessandro
• Is Part Of: Neural networks. Volume 81(2016)
• Page Start: 72
• Page End: 80
• Publication Date: 2016-09
• Subjects: Temporal manifolds -- Regularization networks -- Dissipative systems -- On-line back-propagation; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2016.05.005
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: This paper analyzes the practical issues and reports some results on a theory in which learning is modeled as a continuous temporal process driven by laws describing the interactions of intelligent agents with their own environment. The classic regularization framework is paired with the idea of temporal manifolds by introducing the principle of least cognitive action, which is inspired by the related principle of mechanics. The introduction of the counterparts of the kinetic and potential energy leads to an interpretation of learning as a dissipative process. As an example, we apply the theory to supervised learning in neural networks and show that the corresponding Euler–Lagrange differential equations can be connected to the classic gradient descent algorithm on the supervised pairs. We give preliminary experiments to confirm the soundness of the theory. Highlights: A learning theory based on the variational principle of least cognitive action. Supervised On-line Learning evolving as a dissipative dynamic system. Stochastic or Batch Gradient Descent are obtained by varying the dissipation level. Experimental evaluation on standard and custom benchmarks.