Emerging trends: A gentle introduction to fine-tuning. (November 2021)

• Record Type: Journal Article
• Authors: Church, Kenneth Ward; Chen, Zeyu; Ma, Yanjun
• Is Part Of: Natural language engineering. Volume 27:Part 6(2021)
• Page Start: 763
• Page End: 778
• Publication Date: 2021-11
• Subjects: Deep nets -- Pre-training -- Fine-tuning -- Benchmarks -- GLUE -- SQuAD -- ImageNet -- BERT -- ERNIE -- wav2vec; Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324921000322
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: The previous Emerging Trends article (Church et al ., 2021 . Natural Language Engineering 27 (5), 631–645.) introduced deep nets to poets. Poets is an imperfect metaphor, intended as a gesture toward inclusion. The future for deep nets will benefit by reaching out to a broad audience of potential users, including people with little or no programming skills, and little interest in training models. That paper focused on inference, the use of pre-trained models, as is, without fine-tuning. The goal of this paper is to make fine-tuning more accessible to a broader audience. Since fine-tuning is more challenging than inference, the examples in this paper will require modest programming skills, as well as access to a GPU. Fine-tuning starts with a general purpose base (foundation) model and uses a small training set of labeled data to produce a model for a specific downstream application. There are many examples of fine-tuning in natural language processing (question answering (SQuAD) and GLUE benchmark), as well as vision and speech.