A label-oriented loss function for learning sentence representations. (March 2021)

• Record Type: Journal Article
• Authors: Liu, Yihong; Guan, Wei; Lu, Dongxu; Zou, Xianchun
• Is Part Of: Computer speech & language. Volume 66(2021)
• Publication Date: 2021-03
• Subjects: Label-embeddings -- Label-oriented loss -- Multi-LSTM -- Sentence representations -- Sentiment analysis; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2020.101165
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: We do not take end-to-end approach but utilize a neural network structure combined with multi-LSTM layer and fully connected layers, to encode the initial word-embedding sequence of a sentence to its meaningful sentence representation. Then the downstream classification is performed by KNN classifier. We propose a novel label-oriented loss function, which takes the positive label-embedding and the negative labelembeddings into account. We report attractive performances achieved on several sentiment and sentence classification datasets in our evaluation. Abstract: Neural network methods which leverage word-embedding obtained from unsupervised learning models have been widely adopted in many natural language processing (NLP) tasks, including sentiment analysis and sentence classification. Existing sentence representation generation approaches which serve for classification tasks generally rely on complex deep neural networks but relatively simple loss functions, such as cross entropy loss function. These approaches cannot produce satisfactory separable sentence representations because the usage of cross entropy may ignore the sentiment and semantic information of the labels. To extract useful information from labels for improving the distinguishability of the obtained sentence representations, this paper proposes a label-oriented loss function. The proposed loss function takes advantage of the word-embeddings of labels to guide the production of meaningful sentence representations which serve for downstream classification tasks. Compared with existing end-to-end approaches, the evaluation experiments on several datasets illustrate that using the proposed loss function can achieve competitive and even better classification results.