Transfer fine-tuning of BERT with phrasal paraphrases. (March 2021)

• Record Type: Journal Article
• Authors: Arase, Yuki; Tsujii, Junichi
• Is Part Of: Computer speech & language. Volume 66(2021)
• Publication Date: 2021-03
• Subjects: Transfer fine-tuning -- Paraphrase -- Sentence pair modelling -- Sentence representation -- BERT; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2020.101164
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: Transfer fine-tuning yields representations suitable for specific tasks; in this paper we focused on sentence pair modelling. The method helps the BERT model converge more quickly with a smaller corpus. It also realizes performance gains while maintaining the model size. Simple features outperform elaborate ones in phrasal paraphrase classification. Abstract: Sentence pair modelling is defined as the task of identifying the semantic interaction between a sentence pair, i.e ., paraphrase and textual entailment identification and semantic similarity measurement. It constitutes a set of crucial tasks for research in the area of natural language understanding. Sentence representation learning is a fundamental technology for sentence pair modelling, where the development of the BERT model realised a breakthrough. We have recently proposed transfer fine-tuning using phrasal paraphrases to allow BERT's representations to be suitable for semantic equivalence assessment between sentences while maintaining the model size. Herein, we reveal that transfer fine-tuning with simplified feature generation allows us to generate representations that are widely effective across different types of sentence pair modelling tasks. Detailed analysis confirms that our transfer fine-tuning helps the BERT model converge more quickly with a smaller corpus for fine-tuning.