Stochastic optimal well control in subsurface reservoirs using reinforcement learning. (September 2022)

• Record Type: Journal Article
• Authors: Dixit, Atish; ElSheikh, Ahmed H.
• Is Part Of: Engineering applications of artificial intelligence. Volume 114(2022)
• Publication Date: 2022-09
• Subjects: 0000 -- 1111; Reinforcement learning -- Stochastic optimal control -- Subsurface flow control -- Artificial intelligence in reservoir management -- Optimal control for partially observable system; Engineering -- Data processing -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Ingénierie -- Informatique -- Périodiques; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; Artificial intelligence; Engineering -- Data processing; Expert systems (Computer science); Periodicals; 620.00285
• DOI: 10.1016/j.engappai.2022.105106
• Languages: English
• ISSNs: 0952-1976

Abstract

Abstract: We present a case study of model-free reinforcement learning (RL) framework to solve stochastic optimal control for a predefined parameter uncertainty distribution and partially observable system. We focus on robust optimal well control problem which is a subject of intensive research activities in the field of subsurface reservoir management. For this problem, the system is partially observed since the data is only available at well locations. Furthermore, the model parameters are highly uncertain due to sparsity of available field data. In principle, RL algorithms are capable of learning optimal action policies – a map from states to actions – to maximize a numerical reward signal. In deep RL, this mapping from state to action is parameterized using a deep neural network. In the RL formulation of the robust optimal well control problem, the states are represented by saturation and pressure values at well locations while the actions represent the valve openings controlling the flow through wells. The numerical reward refers to the total sweep efficiency and the uncertain model parameter is the subsurface permeability field. The model parameter uncertainties are handled by introducing a domain randomization scheme that exploits cluster analysis on its uncertainty distribution. We present numerical results using two state-of-the-art RL algorithms, proximal policy optimization (PPO) and advantage actor–critic (A2C), on two subsurface flow test cases representing two distinct uncertainty distributions of permeability field. The results were benchmarked against optimization results obtained using differential evolution algorithm. Furthermore, we demonstrate the robustness of the proposed use of RL by evaluating the learned control policy on unseen samples drawn from the parameter uncertainty distribution that were not used during the training process. Highlights: Reinforcement learning (RL) methodology to solve stochastic optimal control problem Domain randomization method to evaluate robustness of learned optimal policies Two test cases of stochastic optimal well control problem from reservoir management Optimality of RL policies is benchmarked with the differential evolution algorithm