Make That Sound More Metallic: Towards a Perceptually Relevant Control of the Timbre of Synthesizer Sounds Using a Variational Autoencoder. Issue 1 (18th May 2021)

• Record Type: Journal Article
• Authors: Roche, Fanny; Hueber, Thomas; Garnier, Maëva; Limier, Samuel; Girin, Laurent
• Is Part Of: Transactions of the International Society for Music Information Retrieval. Volume 4:Issue 1(2021)
• Page Start: 52
• Page End: 66
• Publication Date: 2021-05-18
• Subjects: Synthesizer sounds -- timbre perception and verbal description -- variational autoencoders -- machine learning -- audio synthesis; 025
• DOI: 10.5334/tismir.76
• Languages: English
• ISSNs: 2514-3298

Abstract

In this article, we propose a new method of sound transformation based on control parameters that are intuitive and relevant for musicians. This method uses a variational autoencoder (VAE) model that is first trained in an unsupervised manner on a large dataset of synthesizer sounds. Then, a perceptual regularization term is added to the loss function to be optimized, and a supervised fine-tuning of the model is carried out using a small subset of perceptually labeled sounds. The labels were obtained from a perceptual test of Verbal Attribute Magnitude Estimation in which listeners rated this training sound dataset along eight perceptual dimensions (French equivalents of metallic, warm, breathy, vibrating, percussive, resonating, evolving, aggressive ). These dimensions were identified as relevant for the description of synthesizer sounds in a first Free Verbalization test. The resulting VAE model was evaluated by objective reconstruction measures and a perceptual test. Both showed that the model was able, to a certain extent, to capture the acoustic properties of most of the perceptual dimensions and to transform sound timbre along at least two of them ( aggressive and vibrating ) in a perceptually relevant manner. Moreover, it was able to generalize to unseen samples even though a small set of labeled sounds was used.