A Decreasing Scaling Transition Scheme from Adam to SGD. Issue 7 (27th April 2022)

• Record Type: Journal Article
• Authors: Zeng, Kun; Liu, Jinlan; Jiang, Zhixia; Xu, Dongpo
• Is Part Of: Advanced theory and simulations. Volume 5:Issue 7(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-04-27
• Subjects: deep learning -- image classification -- learning rate -- stochastic gradient descent; Science -- Simulation methods -- Periodicals; Science -- Methodology -- Periodicals; Engineering -- Simulation methods -- Periodicals; Engineering -- Methodology -- Periodicals; 507.21
• DOI: 10.1002/adts.202100599
• Languages: English
• ISSNs: 2513-0390

Abstract

Abstract: Adaptive gradient algorithm and its variants, such as RMSProp, Adam, AMSGrad, etc., have been widely used in deep learning. Although these algorithms are faster in the early phase of training, their generalization performance is often not as good as stochastic gradient descent (SGD). Hence, a trade‐off method of transforming Adam to SGD after a certain iteration to gain the merits of both algorithms is theoretically and practically significant. To that end, a decreasing scaling transition scheme to achieve a smooth and stable transition from Adam to SGD, which is called DSTAdam. The convergence of the proposed DSTAdam is also proved in an online convex setting. Finally, the effectiveness of the DSTAdam is verified on the different datasets. The implementation is available at: https://github.com/kunzeng/DSTAdam . Abstract : A trade‐off method of transforming Adam to SGD after a certain iteration to gain the merits of both algorithms is theoretically and practically significant. A decreasing scaling transition scheme to achieve a smooth and stable transition from Adam to SGD for the deep learning, which is called DSTAdam, is proposed. DSTAdam has fast convergence speed and good generalization performance.