Active inference and learning. (September 2016)

• Record Type: Journal Article
• Authors: Friston, Karl; FitzGerald, Thomas; Rigoli, Francesco; Schwartenbeck, Philipp; O'Doherty, John; Pezzulo, Giovanni
• Is Part Of: Neuroscience and biobehavioral reviews. Volume 68(2016)
• Page Start: 862
• Page End: 879
• Publication Date: 2016-09
• Subjects: Active inference -- Habit learning -- Bayesian inference -- Goal-directed -- Free energy -- Information gain -- Bayesian surprise -- Epistemic value -- Exploration -- Exploitation; Psychophysiology -- Periodicals; Human behavior -- Periodicals; Animal behavior -- Periodicals; Neurology -- Periodicals; Behavior -- Periodicals; Ethology -- Periodicals; Neurology -- Periodicals; Psychophysiologie -- Périodiques; Comportement humain -- Périodiques; Animaux -- Mœurs et comportement -- Périodiques; Neurologie -- Périodiques; Animal behavior; Human behavior; Neurology; Psychophysiology; Periodicals; Electronic journals; 573.8
• DOI: 10.1016/j.neubiorev.2016.06.022
• Languages: English
• ISSNs: 0149-7634

Abstract

Highlights: Optimal behaviour is quintessentially belief based. Behaviour can be described as optimising expected free energy. Expected free energy entails pragmatic and epistemic value. Habits are learned by observing one's own goal directed behaviour. Habits are then selected online during active inference. Abstract: This paper offers an active inference account of choice behaviour and learning. It focuses on the distinction between goal-directed and habitual behaviour and how they contextualise each other. We show that habits emerge naturally (and autodidactically) from sequential policy optimisation when agents are equipped with state-action policies . In active inference, behaviour has explorative (epistemic) and exploitative (pragmatic) aspects that are sensitive to ambiguity and risk respectively, where epistemic (ambiguity-resolving) behaviour enables pragmatic (reward-seeking) behaviour and the subsequent emergence of habits. Although goal-directed and habitual policies are usually associated with model-based and model-free schemes, we find the more important distinction is between belief-free and belief-based schemes. The underlying (variational) belief updating provides a comprehensive (if metaphorical) process theory for several phenomena, including the transfer of dopamine responses, reversal learning, habit formation and devaluation. Finally, we show that active inference reduces to a classical (Bellman) scheme, in the absence of ambiguity.