Regularizing transformers with deep probabilistic layers. (April 2023)

• Record Type: Journal Article
• Authors: Aguilera, Aurora Cobo; Olmos, Pablo M.; Artés-Rodríguez, Antonio; Pérez-Cruz, Fernando
• Is Part Of: Neural networks. Volume 161(2023)
• Page Start: 565
• Page End: 574
• Publication Date: 2023-04
• Subjects: Natural language processing -- Regularization -- Deep learning -- Transformers -- Variational auto-encoder -- Missing data; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2023.01.032
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Language models (LM) have grown non-stop in the last decade, from sequence-to-sequence architectures to attention-based Transformers. However, regularization is not deeply studied in those structures. In this work, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer. We study its advantages regarding the depth where it is placed and prove its effectiveness in several scenarios. Experimental result demonstrates that the inclusion of deep generative models within Transformer-based architectures such as BERT, RoBERTa, or XLM-R can bring more versatile models, able to generalize better and achieve improved imputation score in tasks such as SST-2 and TREC or even impute missing/noisy words with richer text.