Unsupervised learning of disentangled representations in deep restricted kernel machines with orthogonality constraints. (October 2021)

• Record Type: Journal Article
• Authors: Tonin, Francesco; Patrinos, Panagiotis; Suykens, Johan A.K.
• Is Part Of: Neural networks. Volume 142(2021)
• Page Start: 661
• Page End: 679
• Publication Date: 2021-10
• Subjects: Kernel methods -- Unsupervised learning -- Manifold learning -- Learning disentangled representations; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2021.07.023
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: We introduce Constr-DRKM, a deep kernel method for the unsupervised learning of disentangled data representations. We propose augmenting the original deep restricted kernel machine formulation for kernel PCA by orthogonality constraints on the latent variables to promote disentanglement and to make it possible to carry out optimization without first defining a stabilized objective. After discussing a number of algorithms for end-to-end training, we quantitatively evaluate the proposed method's effectiveness in disentangled feature learning. We demonstrate on four benchmark datasets that this approach performs similarly overall to β -VAE on several disentanglement metrics when few training points are available while being less sensitive to randomness and hyperparameter selection than β -VAE. We also present a deterministic initialization of Constr-DRKM's training algorithm that significantly improves the reproducibility of the results. Finally, we empirically evaluate and discuss the role of the number of layers in the proposed methodology, examining the influence of each principal component in every layer and showing that components in lower layers act as local feature detectors capturing the broad trends of the data distribution, while components in deeper layers use the representation learned by previous layers and more accurately reproduce higher-level features. Highlights: A deep kernel method for unsupervised representation learning is proposed. A constrained optimization problem on the Stiefel manifold is formulated. Experiments are conducted in denoising and in disentangled feature learning. The influence of each principal component in every layer is studied by denoising complex 2D data. In experiments, our approach was less sensitive than β -VAE to randomness and hyperparameter selection.