A data-driven robust optimization algorithm for black-box cases: An application to hyper-parameter optimization of machine learning algorithms. (October 2021)

• Record Type: Journal Article
• Authors: Seifi, Farshad; Azizi, Mohammad Javad; Akhavan Niaki, Seyed Taghi
• Is Part Of: Computers & industrial engineering. Volume 160(2021)
• Publication Date: 2021-10
• Subjects: Robust optimization -- Data-driven optimization -- Black-box optimization -- Gaussian process -- Bayesian optimization -- Hyper-parameter tuning -- Deep learning; Engineering -- Data processing -- Periodicals; Industrial engineering -- Periodicals; 620.00285
• DOI: 10.1016/j.cie.2021.107581
• Languages: English
• ISSNs: 0360-8352

Abstract

Graphical abstract: Highlights: A novel Black-Box data-driven robust optimization approach is proposed. A Gaussian process is used in a Bayesian optimization framework to design the approach. The approach is consistent with the data in a predefined confidence level. A hyper-parameter optimization for deep learning is investigated as an application. The optimal hyper-parameters are robust with respect to noise. Abstract: The huge availability of data in the last decade has raised the opportunity for the better use of data in decision-making processes. The idea of using the existing data to achieve a more coherent reality solution has led to a branch of optimization called data-driven optimization. On the one hand, the presence of uncertain variables in these datasets makes it crucial to design robust optimization methods in this area. On the other hand, in many real-world problems, the closed-form of the objective function is not available and a meta -model based framework is necessary. Motivated by the above points, in this paper a Gaussian process is used in a Bayesian optimization framework to design a method that is consistent with the data in a predefined confidence level. The advantage of the proposed method is that it is computationally tractable in addition to being robust and independent of the objective function's form. As one of the applications of the proposed algorithm, hyper-parameter optimization for deep learning is investigated. The proposed method can help find the optimal hyper-parameters that are robust with respect to noise.