Meta-reinforcement learning for the tuning of PI controllers: An offline approach. (October 2022)

• Record Type: Journal Article
• Authors: McClement, Daniel G.; Lawrence, Nathan P.; Backström, Johan U.; Loewen, Philip D.; Forbes, Michael G.; Gopaluni, R. Bhushan
• Is Part Of: Journal of process control. Volume 118(2022)
• Page Start: 139
• Page End: 152
• Publication Date: 2022-10
• Subjects: Meta-learning -- Deep learning -- Reinforcement learning -- Adaptive control -- Process control -- PID control; Process control -- Periodicals; Fabrication -- Contrôle -- Périodiques; Process control; Periodicals; Electronic journals; 660.281
• DOI: 10.1016/j.jprocont.2022.08.002
• Languages: English
• ISSNs: 0959-1524

Abstract

Abstract: Meta-learning is a branch of machine learning which trains neural network models to synthesize a wide variety of data in order to rapidly solve new problems. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that can be used to tune proportional–integral controllers. Our meta-RL agent has a recurrent structure that accumulates "context" to learn a system's dynamics through a hidden state variable in closed-loop. This architecture enables the agent to automatically adapt to changes in the process dynamics. In tests reported here, the meta-RL agent was trained entirely offline on first order plus time delay systems, and produced excellent results on novel systems drawn from the same distribution of process dynamics used for training. A key design element is the ability to leverage model-based information offline during training in simulated environments while maintaining a model-free policy structure for interacting with novel processes where there is uncertainty regarding the true process dynamics. Meta-learning is a promising approach for constructing sample-efficient intelligent controllers. Highlights: An expert agent recognizes and controls any process in a large class. Offline training leads to sample-efficient online operation and adaptation. The online agent adapts automatically as plant characteristics drift. The agent adjusts PI parameters instead of directly actuating the plant. meta-RL design generalizes to various plant and controller structures.