When will gradient methods converge to max‐margin classifier under ReLU models?. Issue 1 (9th March 2021)

• Record Type: Journal Article
• Authors: Xu, Tengyu; Zhou, Yi; Ji, Kaiyi; Liang, Yingbin
• Is Part Of: Stat. Volume 10:Issue 1(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-03-09
• Subjects: algorithm -- classification -- linear model -- machine learning -- neural network; Statistics -- Periodicals; 519.2
• DOI: 10.1002/sta4.354
• Languages: English
• ISSNs: 2049-1573

Abstract

Abstract : We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable data set. The classifier is described by a non‐linear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show that there can exist spurious asymptotic local minima besides asymptotic global minima. We then show that gradient descent (GD) can converge to either a global or a local max‐margin direction or may diverge from the desired max‐margin direction in a general context. For stochastic gradient descent (SGD), we show that it converges in expectation to either the global or the local max‐margin direction if SGD converges. We further explore the implicit bias of these algorithms in learning a multineuron network under certain stationary conditions and show that the learned classifier maximizes the margins of each sample pattern partition under the ReLU activation.