Interactive visualization for testing Markov Decision Processes: MDPVIS. (April 2017)

• Record Type: Journal Article
• Authors: McGregor, Sean; Buckingham, Hailey; Dietterich, Thomas G.; Houtman, Rachel; Montgomery, Claire; Metoyer, Ronald
• Is Part Of: Journal of visual languages & computing. Volume 39(2017)
• Page Start: 93
• Page End: 106
• Publication Date: 2017-04
• Subjects: Visualization -- Markov decision process -- Testing -- Parameter space analysis -- Wildfire -- Optimization; Visual programming languages (Computer science) -- Periodicals; Visual programming (Computer science) -- Periodicals; Programming languages (Electronic computers) -- Semantics -- Periodicals; Langages de programmation visuelle -- Périodiques; Programmation visuelle -- Périodiques; Langages de programmation -- Sémantique -- Périodiques; Programming languages (Electronic computers) -- Semantics; Visual programming (Computer science); Visual programming languages (Computer science); Periodicals; Electronic journals; 005
• DOI: 10.1016/j.jvlc.2016.10.007
• Languages: English
• ISSNs: 1045-926X

Abstract

Abstract: Markov Decision Processes (MDPs) are a formulation for optimization problems in sequential decision making. Solving MDPs often requires implementing a simulator for optimization algorithms to invoke when updating decision making rules known as policies. The combination of simulator and optimizer are subject to failures of specification, implementation, integration, and optimization that may produce invalid policies. We present these failures as queries for a visual analytic system (MDPVIS ). MDPVIS addresses three visualization research gaps. First, the data acquisition gap is addressed through a general simulator-visualization interface. Second, the data analysis gap is addressed through a generalized MDP information visualization. Finally, the cognition gap is addressed by exposing model components to the user. MDPVIS generalizes a visualization for wildfire management. We use that problem to illustrate MDPVIS and show the visualization's generality by connecting it to two reinforcement learning frameworks that implement many different MDPs of interest in the research community. Abstract : Highlights: Markov decision processes (MDPs) formalize sequential decision optimization problems. Complex simulators often implement MDPs and are subject to a variety of bugs. Interactive visualizations support testing MDPs and optimization algorithms. The first visualization targeting MDP testing, MDPvis, is presented.