Interfacing HDF5 with a scalable object‐centric storage system on hierarchical storage. (9th March 2020)

• Record Type: Journal Article
• Authors: Mu, Jingqing; Soumagne, Jerome; Byna, Suren; Koziol, Quincey; Tang, Houjun; Warren, Richard
• Other Names: Gil‐Costa Veronica guestEditor.; Senger Hermes guestEditor.; Robinson Timothy W. guestEditor.; Thota Abhinav guestEditor.
• Is Part Of: Concurrency and computation. Volume 32:Number 20(2020)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-03-09
• Subjects: data handling -- HDF5 -- large‐scale systems -- memory management -- object‐centric models; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5715
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Object storage technologies that take advantage of multitier storage on HPC systems are emerging. However, to use these technologies at present, applications have to be modified significantly from current I/O libraries. HDF5, a widely used I/O middleware on HPC systems, provides a virtual object layer (VOL) that allows applications to connect to different storage mechanisms transparently without requiring significant code modifications. We recently designed the proactive data containers (PDC) object‐centric storage system that provides the capabilities of transparent, asynchronous, and autonomous data movement taking advantage of multiple storage tiers—a decision that has so far been left upon the user on most current systems. To enable PDC's features through HDF5 without modifying application codes, we have developed an HDF5 VOL connector that interfaces with PDC. We present in this article the connector interface and evaluate its performance on Cori, a Cray XC40 supercomputer located at the National Energy Research Scientific Computing Center (NERSC). Our evaluation demonstrates up to an 8× improvement compared with HDF5 that has the most recent optimizations.