Data-driven model reference control of MIMO vertical tank systems with model-free VRFT and Q-Learning. (February 2018)

• Record Type: Journal Article
• Authors: Radac, Mircea-Bogdan; Precup, Radu-Emil; Roman, Raul-Cristian
• Is Part Of: ISA transactions. Volume 73(2018)
• Page Start: 227
• Page End: 238
• Publication Date: 2018-02
• Subjects: Batch fitted Q-learning -- Model-free optimal control -- Model reference tracking -- Multi input-multi output systems -- Neural networks -- Vertical tank systems -- Virtual reference feedback tuning; Engineering instruments -- Periodicals; Engineering instruments; Periodicals; Electronic journals; 629.805
• DOI: 10.1016/j.isatra.2018.01.014
• Languages: English
• ISSNs: 0019-0578

Abstract

Abstract: This paper proposes a combined Virtual Reference Feedback Tuning–Q-learning model-free control approach, which tunes nonlinear static state feedback controllers to achieve output model reference tracking in an optimal control framework. The novel iterative Batch Fitted Q-learning strategy uses two neural networks to represent the value function (critic) and the controller (actor), and it is referred to as a mixed Virtual Reference Feedback Tuning–Batch Fitted Q-learning approach. Learning convergence of the Q-learning schemes generally depends, among other settings, on the efficient exploration of the state-action space. Handcrafting test signals for efficient exploration is difficult even for input-output stable unknown processes. Virtual Reference Feedback Tuning can ensure an initial stabilizing controller to be learned from few input-output data and it can be next used to collect substantially more input-state data in a controlled mode, in a constrained environment, by compensating the process dynamics. This data is used to learn significantly superior nonlinear state feedback neural networks controllers for model reference tracking, using the proposed Batch Fitted Q-learning iterative tuning strategy, motivating the original combination of the two techniques. The mixed Virtual Reference Feedback Tuning–Batch Fitted Q-learning approach is experimentally validated for water level control of a multi input-multi output nonlinear constrained coupled two-tank system. Discussions on the observed control behavior are offered. Graphical abstract: Control system with direct Multi Input-Multi Output (MIMO) Batch fitted Q-learning-based controller (black), SISO Virtual Reference Feedback Tuning controllers (magenta) and model-based Value Iteration MIMO controller (blue): a) control inputs u 1 ; b) y 1 d (red); c) control inputs u 2 ; d) y 2 d (red). Highlights: VRFT tunes pre-stabilizing PID controllers for input-state transitions collection. Offline batch fitted Q-learning of a state feedback neural network controller. Learned controllers are validated on a MIMO vertical tank system. Comparisons with initial PID controller and model-based Value Iteration one. Implementation details and insightful comments are offered.