The unexpected virtue of almost: Exploiting MPI collective operations to approximately coordinate checkpoints. (9th September 2018)

• Record Type: Journal Article
• Authors: Levy, Scott; Ferreira, Kurt B.; Widener, Patrick
• Other Names: Jin Hai guestEditor.; Shen Xipeng guestEditor.; Lovas Robert guestEditor.; Liao Xiaofei guestEditor.; Skjellum Anthony guestEditor.; Bangalore Purushotham V. guestEditor.; Grant Ryan E. guestEditor.
• Is Part Of: Concurrency and computation. Volume 32:Number 3(2020)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2018-09-09
• Subjects: checkpoint/restart -- fault tolerance -- MPI; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.4890
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Coordinated checkpoint/restart is currently the dominant approach to mitigating the impact of failures on important scientific applications running on large‐scale distributed systems. However, there is widespread evidence that coordinated checkpointing may no longer be viable on next‐generation systems. Uncoordinated checkpoint/restart attempts to address the shortcomings of coordinated checkpoint/restart by allowing application processes to checkpoint their state independently. However, eliminating coordination may significantly degrade application performance. In this paper, we propose an approach that leverages existing coordination in important scientific applications to approximately coordinate checkpoints. Specifically, we propose to extend MPI implementations to force checkpoints to occur immediately after the completion of a collective operation. We evaluate the performance implications of this approach using an existing validated simulation framework. Our results demonstrate that approximately coordinated checkpointing can significantly improve application performance relative to totally uncoordinated checkpointing. We also show that forcing checkpoints to occur following a collective operation has a small impact on the nominal checkpoint interval for several important workloads. As a whole, the results presented in this paper demonstrate that approximately coordinated checkpointing may provide significant performance benefits without significantly increasing the cost of failure recovery.