Lustre Lockahead: Early experience and performance using optimized locking. (29th September 2017)

• Record Type: Journal Article
• Authors: Moore, Michael; Farrell, Patrick; Cernohous, Bob
• Other Names: Xhafa Fatos guestEditor.; Michael Scott guestEditor.; He Yun guestEditor.
• Is Part Of: Concurrency and computation. Volume 30:Number 1(2018)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2017-09-29
• Subjects: file systems -- Lustre -- MPI‐IO -- LDLM -- WRF; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.4332
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Recent Cray‐authored Lustre modifications known as Lustre Lockahead show significantly improved write performance for collective, shared‐file I/O workloads. Initial tests show write performance improvements of more than 200% for small transfer sizes and over 100% for larger transfer sizes compared to traditional Lustre locking. Standard Lustre shared‐file locking mechanisms limit scaling of shared‐file I/O performance on modern high‐performance Lustre servers. The new Lockahead feature provides a mechanism for applications (or libraries) with knowledge of their I/O patterns to overcome this limitation by explicitly requesting locks. MPI‐IO is able to use this feature to dramatically improve shared‐file collective I/O performance, achieving more than 80% of file per process performance. This paper discusses our early experience using Lockahead with applications. We also present application and synthetic performance results and discuss performance considerations for applications that benefit from Lockahead.