Data-driven model reference control of MIMO vertical tank systems with model-free VRFT and Q-Learning. (February 2018) - Radac, Mircea-Bogdan; Precup, Radu-Emil; Roman, Raul-Cristian
Abstract: This paper proposes a combined Virtual Reference Feedback Tuning–Q-learning model-free control approach, which tunes nonlinear static state feedback controllers to achieve output model reference tracking in an optimal control framework. The novel iterative Batch Fitted Q-learning strategy uses two neural networks to represent the value function (critic) and the controller (actor), and it is referred to as a mixed Virtual Reference Feedback Tuning–Batch Fitted Q-learning approach. Learning convergence of the Q-learning schemes generally depends, among other settings, on the efficient exploration of the state-action space. Handcrafting test signals for efficient exploration is difficult even for input-output stable unknown processes. Virtual Reference Feedback Tuning can ensure an initial stabilizing controller to be learned from few input-output data and it can be next used to collect substantially more input-state data in a controlled mode, in a constrained environment, by compensating the process dynamics. This data is used to learn significantly superior nonlinear state feedback neural networks controllers for model reference tracking, using the proposed Batch Fitted Q-learning iterative tuning strategy, motivating the original combination of the two techniques. The mixed Virtual Reference Feedback Tuning–Batch Fitted Q-learning approach is experimentally validated for water level control of a multi input-multi output nonlinear constrained coupled two-tank system. Discussions on the observed control behavior are offered. Graphical abstract: Control system with direct Multi Input-Multi Output (MIMO) Batch fitted Q-learning-based controller (black), SISO Virtual Reference Feedback Tuning controllers (magenta) and model-based Value Iteration MIMO controller (blue): a) control inputs u 1 ; b) y 1 d (red); c) control inputs u 2 ; d) y 2 d (red). Highlights: VRFT tunes pre-stabilizing PID controllers for input-state transitions collection. Offline batch fitted Q-learning of a state feedback neural network controller. Learned controllers are validated on a MIMO vertical tank system. Comparisons with initial PID controller and model-based Value Iteration one. Implementation details and insightful comments are offered.