Universal characteristics of deep neural network loss surfaces from random matrix theory. (8th December 2022)

• Record Type: Journal Article
• Authors: Baskerville, Nicholas P; Keating, Jonathan P; Mezzadri, Francesco; Najnudel, Joseph; Granziol, Diego
• Is Part Of: Journal of physics. Volume 55:Number 49(2022)
• Publication Date: 2022-12-08
• Subjects: random matrix theory -- neural networks -- universality -- loss surfaces; Mathematical physics -- Periodicals; Statistical physics -- Periodicals; Quantum theory -- Periodicals; Matter -- Properties -- Periodicals; 530.105
• DOI: 10.1088/1751-8121/aca7f5
• Languages: English
• ISSNs: 1751-8113

Abstract

Abstract: This paper considers several aspects of random matrix universality in deep neural networks (DNNs). Motivated by recent experimental work, we use universal properties of random matrices related to local statistics to derive practical implications for DNNs based on a realistic model of their Hessians. In particular we derive universal aspects of outliers in the spectra of deep neural networks and demonstrate the important role of random matrix local laws in popular pre-conditioning gradient descent algorithms. We also present insights into DNN loss surfaces from quite general arguments based on tools from statistical physics and random matrix theory.