Speech and multilingual natural language framework for speaker change detection and diarization. (1st March 2023)

• Record Type: Journal Article
• Authors: Anidjar, Or Haim; Estève, Yannick; Hajaj, Chen; Dvir, Amit; Lapidot, Itshak
• Is Part Of: Expert systems with applications. Volume 213:Part C(2023)
• Publication Date: 2023-03-01
• Subjects: Speaker change detection -- Transformers -- Speech recognition -- Speaker diarization -- Speaker embedding; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.119238
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Speaker Change Detection (SCD) is the problem of splitting an audio-recording by its speaker-turns. Many real-world problems, such as the Speaker Diarization (SD) or automatic speech transcription, are influenced by the quality of the speaker-turns estimation. Previous works have already shown that auxiliary textual information (for mono-lingual systems) can be of great use for detection of speaker-turns and the diarization systems' performance. In this paper, we suggest a framework for speaker-turn estimation, as well as the determination of clustered speaker identities to the SD system, and examine our approach over a multi-lingual dataset that consists of three mono-lingual datasets—in English, French, and Hebrew. As such, we propose a generic and language-independent framework for the SCD problem that is learned through textual information using state-of-the-art transformer-based techniques and speech-embedding modules. Comprehensive experimental evaluation shows that (i) our multi-lingual SCD framework is competitive enough when compared to a framework over mono-lingual datasets, and that (ii) textual information improves the solution's quality compared to the speech signal-based approach. In addition, we show that our multi-lingual SCD approach does not harm the performance of SD systems. Highlights: Improvement of speaker change detection & diarization system by textual data. Multilingual training framework to compute speaker-turns in multiple languages. Competitive strategy transformers and speech embedding for speaker labeling. Additive approach for multilingual domain adaptation in NLP.