Transfer learning meets sales engagement email classification: Evaluation, analysis, and strategies. (6th April 2020)

• Record Type: Journal Article
• Authors: Liu, Yong; Dmitriev, Pavel; Huang, Yifei; Brooks, Andrew; Dong, Li; Liang, Mengyue; Boshernitzan, Zvi; Cao, Jiwei; Nguy, Bobby
• Is Part Of: Concurrency and computation. Volume 34:Number 8(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-04-06
• Subjects: cross‐org transfer learning -- domain shift -- email intent classification -- pre‐trained language model -- sales engagement -- transfer learning; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5759
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: Enterprise email classification in the sales engagement platform is a challenge due to its evolving asynchronous conversational context during the sales process and differences across industries and organizations. This is further exacerbated by the limited amount of labeled emails due to security and privacy constraints. The leaderboard success of using pretrained language models (LMs) such as BERT and various transfer learning techniques promises a paradigm shift to natural language processing, yet the recipe for applying high performance transfer learning (HPTL) in practical applications remains unclear. This article investigates applying HPTL to sales engagement email classification through a series of experiments and analysis. The experiment datasets include two different organizations' emails. The contribution of this paper is 4‐fold: (a) analysis and characterization of the email corpora from different organizations; (b) identification of the best combinations of pre‐trained LMs under different modeling architectures; (c) study of the impact and trade‐off of limited labeled data on the model accuracy and training time; and (d) characterization and study of the impact of different orgs' datasets on the model accuracy. Our results showed that a practical winning recipe that uses BERT‐finetuning with as few as 500 labeled training examples can consistently outperform significantly with reasonable training time among all models evaluated.