Applying unweighted least‐squares based techniques to stochastic dynamic programming: theory and application. Issue 15 (1st August 2019)

• Record Type: Journal Article
• Authors: Forootani, Ali; Iervolino, Raffaele; Tipaldi, Massimo
• Is Part Of: IET control theory & applications. Volume 13:Issue 15(2019)
• Page Start: 2387
• Page End: 2398
• Publication Date: 2019-08-01
• Subjects: mathematics computing -- stochastic processes -- function approximation -- Markov processes -- data analysis -- probability -- Big Data -- resource allocation -- dynamic programming -- least squares approximations; stochastic dynamic programming -- automatic control system society -- square projection approach -- DP optimisation problem -- monotonicity properties -- DP algorithm -- value function approximation -- uniform probability distribution -- unweighted LS sub‐problem -- Markov decision process model -- resource allocation problem -- MATLAB based software -- big data analysis -- contraction mapping -- unweighted least‐squares based techniques; Control theory -- Periodicals; Automatic control -- Periodicals; 629.8312
• DOI: 10.1049/iet-cta.2019.0289
• Languages: English
• ISSNs: 1751-8644

Abstract

Abstract : Big data and the curse of dimensionality are common vocabularies that researchers in different communities have recently been dealing with, e.g. dynamic programming (DP) in automatic control system society. A novel unweighted sampled based least square projection approach is proposed in this study to address the issue of the large state space in the DP optimisation problem. The method, in particular, takes into account both contraction mapping and monotonicity properties of the DP algorithm for value function approximation. Specifically, the batch of samples are gathered by uniform probability distribution at first, and an unweighted LS sub‐problem in the subspace is solved. As the case study, a new Markov decision process model associated with a resource allocation problem is considered to illustrate the technique and evaluate its effectiveness. It is noted that the approach can be employed for different applications as well. Moreover, a MATLAB based software is developed to implement and examine different parts of the proposed method. Simulation examples are considered to support the results of the approach via developed software. The idea makes a connection between the recent advances in big data analysis and approximate DP as well.