BERT syntactic transfer: A computational experiment on Italian, French and English languages. (January 2022)

• Record Type: Journal Article
• Authors: Guarasci, Raffaele; Silvestri, Stefano; De Pietro, Giuseppe; Fujita, Hamido; Esposito, Massimo
• Is Part Of: Computer speech & language. Volume 71(2022)
• Publication Date: 2022-01
• Subjects: Cross language -- Dependency Parse Tree -- Language models -- Multilingual BERT -- Transfer learning -- Syntactic phenomena; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2021.101261
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract: This paper investigates the ability of multilingual BERT (mBERT) language model to transfer syntactic knowledge cross-lingually, verifying if and to which extent syntactic dependency relationships learnt in a language are maintained in other languages. In detail, the main contributions of this paper are: (i) an analysis of the cross-lingual syntactic transfer capability of mBERT model; (ii) a detailed comparison of cross-language syntactic transfer among languages belonging to different branches of the Indo-European languages, namely English, Italian and French, which present very different syntactic constructions; (iii) a study on the transferability of a syntactic phenomenon peculiar of Italian language, namely the pronoun dropping (pro-drop), also known as omissibility of the subject. To this end, a structural probe devoted to reconstruct the dependency parse tree of a sentence has been exploited, representing the input sentences with the contextual embeddings from mBERT layers. The results of the experimental assessment have shown a transfer of syntactic knowledge of the mBERT model among these languages. Moreover, the behaviour of the probe in the transition from pro-drop to non-pro-drop languages and vice versa has proven to be more effective in case of languages sharing a common linguistic matrix. The possibility of transferring syntactical knowledge, especially in the case of specific phenomena, meets both a theoretical need and can have important practical implications in syntactic tasks, such as dependency parsing. Highlights: An analysis of cross-lingual syntax transfer capability of multilingual BERT model. Use of a structural probe to reconstruct the dependency parse tree of a sentence. A cross-lingual test on an aligned and parallel annotated treebank. Three different languages considered: English, Italian and French. Study of cross-language transfer of omissibility of the subject syntactic phenomenon.