Deep multi-agent reinforcement learning for multi-level preventive maintenance in manufacturing systems. (15th April 2022)

• Record Type: Journal Article
• Authors: Su, Jianyu; Huang, Jing; Adams, Stephen; Chang, Qing; Beling, Peter A.
• Is Part Of: Expert systems with applications. Volume 192(2022)
• Publication Date: 2022-04-15
• Subjects: Multi-level preventive maintenance -- Deep multi-agent reinforcement learning -- Deep reinforcement learning -- Serial production line -- Production loss; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2021.116323
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Designing preventive maintenance (PM) policies that ensure smooth and efficient production for large-scale manufacturing systems is non-trivial. Recent model-free reinforcement learning (RL) methods shed lights on how to cope with the non-linearity and stochasticity in such complex systems. However, the action space explosion impedes RL-based PM policies to be generalized to real applications. In order to obtain cost efficient PM policies for a serial production line that has multiple levels of PM actions, a novel multi-agent modeling is adopted to support adaptive learning by modeling each machine as cooperative agent. The evaluation of system-level production loss is leveraged to construct the reward function. An adaptive learning framework based on value-decomposition multi-agent actor–critic algorithm is utilized to obtain PM policies. In simulation study, the proposed framework demonstrates its effectiveness by leading other baselines on a comprehensive set of metrics whereas the centralized RL-based methods struggles to converge to stable policies. Our analysis further demonstrates that our multi-agent reinforcement learning based method learns effective PM policies without any knowledge about the environment and maintenance strategies. Highlights: Apply Deep Multi-Agent Reinforcement Learning to PM problem in manufacturing system. Incorporate multiple levels of PM actions with imperfect maintenance effects. Avoid space explosion by using Decentralized Partially Observable MDP (Dec-POMDP). Demonstrate the proposed method by applying it to manufacturing applications.