Scaling Identifiers and their Metadata to Gigascale: An Architecture to Tackle the Challenges of Volume and Variety. (1st March 2023)

• Record Type: Journal Article
• Authors: Klump, Jens; Fils, Doug; Devaraju, Anusuriya; Ramdeen, Sarah; Robertson, Jess; Wyborn, Lesley; Lehnert, Kerstin
• Is Part Of: Data science journal. Volume 22(2023)
• Publication Date: 2023-03-01
• Subjects: persistent identifiers -- metadata -- web architecture -- linked open data; Science -- Data processing -- Periodicals; Database management -- Periodicals; 502.85
• DOI: 10.5334/dsj-2023-005
• Languages: English
• ISSNs: 1683-1470

Abstract

Persistent identifiers are applied to an ever-increasing variety of research objects, including software, samples, models, people, instruments, grants, and projects, and there is a growing need to apply identifiers at a finer and finer granularity. Unfortunately, the systems developed over two decades ago to manage identifiers and the metadata describing the identified objects no longer scale. Communities working with physical samples have grappled with these three challenges of the increasing volume, variety, and variability of identified objects for many years. To address this dual challenge, the IGSN 2040 project explored how metadata and catalogues for physical samples could be shared at the scale of billions of samples across an ever-growing variety of users and disciplines. In this paper, we focus on how we scale identifiers and their describing metadata to billions of objects and who the actors involved with this system are. Our analysis of these requirements resulted in the definition of a minimum viable product and the design of an architecture that not only addresses the challenges of increasing volume and variety but, more importantly, is easy to implement because it reuses commonly used Web components. Our solution is based on a Web architectural model that utilisesSchema.org, JSON-LD, and sitemaps. Applying these commonly used architectural patterns on the internet allows us to not only handle increasing variety but also enable better compliance with the FAIR Guiding Principles.