A free-energy principle for representation learning. Issue 4 (15th July 2021)

• Record Type: Journal Article
• Authors: Gao, Yansong; Chaudhari, Pratik
• Is Part Of: Machine learning: science and technology. Volume 2:Issue 4(2021)
• Publication Date: 2021-07-15
• Subjects: information theory -- thermodynamics -- rate-distortion theory -- transfer learning -- information bottleneck; 006.31
• DOI: 10.1088/2632-2153/abf984
• Languages: English
• ISSNs: 2632-2153

Abstract

Abstract: This paper employs a formal connection of machine learning with thermodynamics to characterize the quality of learned representations for transfer learning. We discuss how information-theoretic functionals such as rate, distortion and classification loss of a model lie on a convex, so-called, equilibrium surface. We prescribe dynamical processes to traverse this surface under specific constraints; in particular we develop an iso-classification process that trades off rate and distortion to keep the classification loss unchanged. We demonstrate how this process can be used for transferring representations from a source task to a target task while keeping the classification loss constant. Experimental validation of the theoretical results is provided on image-classification datasets.