RMAML: Riemannian meta-learning with orthogonality constraints. (August 2023)

• Record Type: Journal Article
• Authors: Tabealhojeh, Hadi; Adibi, Peyman; Karshenas, Hossein; Roy, Soumava Kumar; Harandi, Mehrtash
• Is Part Of: Pattern recognition. Volume 140(2023)
• Publication Date: 2023-08
• Subjects: Meta-learning -- Geometry-aware optimization -- Riemannian manifolds -- Few-shot image classification; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109563
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Presenting a meta-learning method based on Riemannian bi-level optimization. Orthogonality constraints on the parameters of neural network layers. More stable meta-learning and higher performance. Higher performance using smaller number of inner-level steps and/or meta-BatchSize. Outperforming the Euclidean counterpart method in few-shot classification. Abstract: Meta-learning is the core capability that enables intelligent systems to rapidly generalize their prior experience to learn new tasks. In general, the optimization-based methods formalize the meta-learning as a bi-level optimization problem, that is a nested optimization framework, in which meta-parameters are optimized (or learned) at the outer-level, while the inner-level optimizes the task-specific parameters. In this paper, we introduce RMAML, a meta-learning method that enforces orthogonality constraints to the bi-level optimization problem. We develop a geometry aware framework that generalizes the bi-level optimization problem to the Riemannian (constrained) setting. Using the Riemannian operations such as orthogonal projection, retraction and parallel transport, the bi-level optimization is reformulated so that it respects the Riemannian geometry. Moreover, we observe that a superior stable optimization and an improved generalization ability can be achieved when the parameters and meta-parameters of the method are modeled using a Stiefel Manifold. We empirically show that RMAML can easily reach competitive performances against several state of the art algorithms for few-shot classification and consistently outperforms its Euclidean counterpart, MAML. For example, by using the geometry of the Stiefel manifold to structure the fully-connected layers in a deep neural network, a 7% increase in single-domain few-shot classification accuracy is achieved. For the cross-domain few-shot learning, RMAML outperforms MAML by up to 9% of accuracy. Our ablation study also demonstrates the effectiveness of RMAML over MAML in terms of higher accuracy with a reduced number of tasks and (or) inner-level updates.