A Sarsa(λ) Algorithm Based on Double-Layer Fuzzy Reasoning. (5th December 2013)

• Record Type: Journal Article
• Authors: Liu, Quan; Mu, Xiang; Huang, Wei; Fu, Qiming; Zhang, Yonggang
• Other Names: Ma Zhiqiang Academic Editor.
• Is Part Of: Mathematical problems in engineering. Volume 2013(2013)
• Publication Date: 2013-12-05
• Subjects: Engineering mathematics -- Periodicals; 510.2462
• DOI: 10.1155/2013/561026
• Languages: English
• ISSNs: 1024-123X

Abstract

Abstract : Solving reinforcement learning problems in continuous space with function approximation is currently a research hotspot of machine learning. When dealing with the continuous space problems, the classic Q -iteration algorithms based on lookup table or function approximation converge slowly and are difficult to derive a continuous policy. To overcome the above weaknesses, we propose an algorithm named DFR-Sarsa( λ ) based on double-layer fuzzy reasoning and prove its convergence. In this algorithm, the first reasoning layer uses fuzzy sets of state to compute continuous actions; the second reasoning layer uses fuzzy sets of action to compute the components of Q -value. Then, these two fuzzy layers are combined to compute the Q -value function of continuous action space. Besides, this algorithm utilizes the membership degrees of activation rules in the two fuzzy reasoning layers to update the eligibility traces. Applying DFR-Sarsa( λ ) to the Mountain Car and Cart-pole Balancing problems, experimental results show that the algorithm not only can be used to get a continuous action policy, but also has a better convergence performance.