A bi-directional attention guided cross-modal network for music based dance generation. (October 2022)

• Record Type: Journal Article
• Authors: Fan, Di; Wan, Lili; Xu, Wanru; Wang, Shenghui
• Is Part Of: Computers & electrical engineering. Volume 103(2022)
• Publication Date: 2022-10
• Subjects: AI choreography -- Cross-modal learning -- Bi-directional attention -- Scheduled sampling; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2022.108310
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: AI choreography has been an active task over the past few years. A fundamental problem of generating dance from music is to effectively learn the correlation between two modalities and then to create long-term body movements. To bridge the gap between music and motion, we propose a novel transformer-based cross-modal framework with a bi-directional attention module, aiming at generating high-quality dance sequences according to the music. The bi-directional attention module contains two layers: one is from music to dance, and the other is from dance to music, bringing benefits to the information exchange and alignment between two modalities. Furthermore, to alleviate exposure bias in sequence-to-sequence generation, we exploit a scheduled sampling strategy that is compatible with the transformer, which reduces error accumulation in long dance generation. Experimental results on 3D dance datasets demonstrate that our approach can achieve rhythmic natural long dance sequences. Graphical abstract: Highlights: A novel transformer-based music to dance framework is proposed. A bi-directional attention decoder effectively fuses music and pose features. A scheduled sampling strategy helps to alleviate the exposure bias.