A Case for Reproducibility in MIR: Replication of 'A Highly Robust Audio Fingerprinting System'. Issue 1 (4th September 2018)

• Record Type: Journal Article
• Authors: Six, Joren; Bressan, Federica; Leman, Marc
• Is Part Of: Transactions of the International Society for Music Information Retrieval. Volume 1:Issue 1(2018)
• Page Start: 56
• Page End: 67
• Publication Date: 2018-09-04
• Subjects: Replication -- Acoustic fingerprinting -- Reproducibility; 025
• DOI: 10.5334/tismir.4
• Languages: English
• ISSNs: 2514-3298

Abstract

Claims made in many Music Information Retrieval (MIR) publications are hard to verify due to the fact that (i) often only a textual description is made available and code remains unpublished – leaving many implementation issues uncovered; (ii) copyrights on music limit the sharing of datasets; and (iii) incentives to put effort into reproducible research – publishing and documenting code and specifics on data – is lacking. In this article the problems around reproducibility are illustrated by replicating an MIR work. The system and evaluation described in 'A Highly Robust Audio Fingerprinting System' is replicated as closely as possible. The replication is done with several goals in mind: to describe difficulties in replicating the work and subsequently reflect on guidelines around reproducible research. Added contributions are the verification of the reported work, a publicly available implementation and an evaluation method that is reproducible.