Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. (23rd August 2013)

• Record Type: Journal Article
• Authors: Liu, Qing; Logan, Jeremy; Tian, Yuan; Abbasi, Hasan; Podhorszki, Norbert; Choi, Jong Youl; Klasky, Scott; Tchoua, Roselyne; Lofstead, Jay; Oldfield, Ron; Parashar, Manish; Samatova, Nagiza; Schwan, Karsten; Shoshani, Arie; Wolf, Matthew; Wu, Kesheng; Yu, Weikuan
• Is Part Of: Concurrency and computation. Volume 26:Number 7(2014:May)
• Page Start: 1453
• Page End: 1473
• Publication Date: 2013-08-23
• Subjects: Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3125
• Languages: English
• ISSNs: 1532-0626

Abstract

<abstract abstract-type="main" id="cpe3125-abs-0001"> <title>SUMMARY</title> <p id="cpe3125-para-0001">Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era. Copyright © 2013 John Wiley &amp; Sons, Ltd.</p> </abstract>