An Actor-Critic Method for Simulation-Based Optimization. Issue 11 (2022)

• Record Type: Journal Article
• Authors: Li, Kuo; Jia, Qing-Shan; Yan, Jiaqi
• Is Part Of: IFAC-PapersOnLine. Volume 55:Issue 11(2022)
• Page Start: 7
• Page End: 12
• Publication Date: 2022
• Subjects: simulation-based optimization -- reinforcement learning -- policy searching -- actor-critic -- policy iteration; Automatic control -- Periodicals; 629.805
• DOI: 10.1016/j.ifacol.2022.08.040
• Languages: English
• ISSNs: 2405-8963

Abstract

Abstract: In this work, we study simulation-based optimization, where the agent aims to select the best configuration from the design space with as few as possible iterations. Inspired by the success of deep reinforcement learning (DRL), we formulate the sampling process as policy searching and give a solving method from the perspective of policy iteration. Concretely, a surrogate model for predicting the performance of each configuration and a parameterized sampling policy are applied, which correspond to the critic and actor in actor-critic (AC) method, respectively. We further derive the updating rule and propose two algorithms for configuration selection in continuous and discrete design spaces, respectively. Finally, the algorithms are validated experimentally on 1) two toy examples to intuitively explain the principle and 2) two high-dimensional tasks to reveal the effectiveness in large-scale problems. The results show that the proposed algorithms can efficiently deal with large-scale problems and effectively eliminate sub-optimal configurations.