An efficient modified Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization. Issue 2 (1st February 2022)

• Record Type: Journal Article
• Authors: Lin, Jingliang; Li, Haiyan; Huang, Yunbao; Chen, Jinghuan; Huang, Pengcheng; Huang, Zeying
• Is Part Of: Engineering optimization. Volume 54:Issue 2(2022)
• Page Start: 252
• Page End: 268
• Publication Date: 2022-02-01
• Subjects: Hyperparameter optimization -- Hyperband -- mode-pursuing sampling -- deep learning -- trust region; Engineering design -- Periodicals; Mathematical optimization -- Periodicals; 620.0042
• DOI: 10.1080/0305215X.2020.1862823
• Languages: English
• ISSNs: 0305-215X

Abstract

Abstract : Although deep learning algorithms have been widely used, their performance depends heavily on a good set of hyperparameters. This article presents an efficient Hyperband and trust-region-based mode-pursuing sampling hybrid method for hyperparameter optimization. First, Hyperband is modified and used to select the optimum quickly from a large number of random sampling points to construct a trust region. Secondly, mode-pursuing sampling is performed in the trust region to generate more points systematically around the minimum, and the location or size of the trust region is dynamically adjusted to accelerate its convergence. Thirdly, the process of selection and sampling is repeated until a termination criterion is met. Numerical examples are presented to verify the effectiveness of the hybrid method, the results of which are compared with those of five well-known algorithms. Comparison results show that better optimal solutions are obtained through the hybrid method, with a higher efficiency.