A Framework for pre-training hidden-unit conditional random fields and its extension to long short term memory networks. (November 2017)

• Record Type: Journal Article
• Authors: Kim, Young-Bum; Stratos, Karl; Sarikaya, Ruhi
• Is Part Of: Computer speech & language. Volume 46(2017)
• Page Start: 311
• Page End: 326
• Publication Date: 2017-11
• Subjects: Pre-training -- Transfer learning -- Spoken language understanding -- Sequence labeling -- Conditional random fiends -- Multi-sense clustering -- Word embedding -- Hidden unit conditional random fields -- LSTMs; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2017.05.004
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract: In this paper, we introduce a simple unsupervised framework for pre-training hidden-unit conditional random fields (HUCRFs), i.e., learning initial parameter estimates for HUCRFs prior to supervised training.Our framework exploits the model structure of HUCRFs to make effective use of unlabeled data from the same domain or labeled data from a different domain. The key idea is to use the separation of HUCRF parameters between observations and labels: this allows us to pre-train observation parameters independently of label parameters. Pre-training is achieved by creating pseudo-labels from such resources. In the case of unlabeled data, we cluster observations and use the resulting clusters as pseudo-labels. Observation parameters can be trained on these resources and then transferred to initialize the supervised training process on the target labeled data. Experiments on various sequence labeling tasks demonstrate that the proposed pre-training method consistently yields significant improvement in performance. The core idea could be extended to other learning techniques including deep learning. We applied the proposed technique to recurrent neural networks (RNN) with long short term memory (LSTM) architecture and obtained similar gains.