DRL-S: Toward safe real-world learning of dynamic thermal management in data center. (15th March 2023)

• Record Type: Journal Article
• Authors: Zhang, Qingang; Chng, Chin-Boon; Chen, Kaiqi; Lee, Poh-Seng; Chui, Chee-Kong
• Is Part Of: Expert systems with applications. Volume 214(2023)
• Publication Date: 2023-03-15
• Subjects: Data Center -- Constrained Deep Reinforcement Learning -- Dynamic Thermal Management -- Data-Driven Model; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.119146
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: Deep Reinforcement Learning considers soft and hard constraint violations. Data-driven approaches avoid explicit descriptions of nonlinear and dynamic systems. Shielding projects unsafe actions into safe action spaces without affecting learning. Power consumption of data centers is optimized while satisfying thermal constraints. Abstract: Deep Reinforcement Learning has been researched for Dynamic Thermal Management in Data Centers. An objective of Dynamic Thermal Management is to minimize power consumption while satisfying a set of hard and soft constraints. However, there is a lack of research on the safety issues of applying Deep Reinforcement Learning in Data Centers during real-world learning, limiting its deployment. To this end, this paper proposes a new method named DRL-S, which can solve both hard and soft constraints simultaneously during the learning stage. In addition to optimizing the main objective, Lagrangian-based Constrained Deep Reinforcement Learning and Reward Shaping enforce policies to satisfy soft constraints through extensive online learning. However, the policy typically fails to satisfy the hard constraints due to the random sampling of actions to encourage exploration and the imperfect policy at the initial online learning stage. To ensure hard constraints are satisfied, we further propose to utilize parameterized Shielding, integrating the approximation of the system dynamics and the projection of the action space to predict the safety of candidate actions and provide backup actions when necessary. Results show that the Lagrangian-based method and Reward Shaping can gradually learn policies to reduce soft constraint violations. The former can better balance the relationship between the main objective and violations by updating Lagrangian multipliers. DRL-S can also effectively avoid extreme temperatures without affecting the normal learning process of vanilla algorithms. The asymptotic power consumption is more than 12% lower than the baseline controller.