Recommendations to Improve Downloads of Large Earth Observation Data. (24th January 2018)

• Record Type: Journal Article
• Authors: Ramachandran, Rahul; Lynnes, Christopher; Baynes, Kathleen; Murphy, Kevin; Baker, Jamie; Kinney, Jamie; Gold, Ariel; Sundwall, Jed; Korver, Mark; Lieber, Allison; Vambenepe, William; Hancher, Matthew; Moore, Rebecca; Erickson, Tyler; Henretig, Josh; Zwiefel, Brant; Patrick-Ahlstrom, Heather; Smith, Matthew J.
• Is Part Of: Data science journal. Volume 17(2018)
• Publication Date: 2018-01-24
• Subjects: Earth Observation Data -- Large Data Transfers -- Cloud -- Best Practices; Science -- Data processing -- Periodicals; Database management -- Periodicals; 502.85
• DOI: 10.5334/dsj-2018-002
• Languages: English
• ISSNs: 1683-1470

Abstract

With the volume of Earth observation data expanding rapidly, cloud computing is quickly changing the way these data are processed, analyzed, and visualized. Collocating freely available Earth observation data on a cloud computing infrastructure may create opportunities unforeseen by the original data provider for innovation and value-added data re-use, but existing systems at data centers are not designed for supporting requests for large data transfers. A lack of common methodology necessitates that each data center handle such requests from different cloud vendors differently. Guidelines are needed to support enabling all cloud vendors to utilize a common methodology for bulk-downloading data from data centers, thus preventing the providers from building custom capabilities to meet the needs of individual vendors. This paper presents recommendations distilled from use cases provided by three cloud vendors (Amazon, Google, and Microsoft) and are based on the vendors' interactions with data systems at different Federal agencies and organizations. These specific recommendations range from obvious steps for improving data usability (such as ensuring the use of standard data formats and commonly supported projections) to non-obvious undertakings important for enabling bulk data downloads at scale. These recommendations can be used to evaluate and improve existing data systems for high-volume data transfers, and their adoption can lead to cloud vendors utilizing a common methodology.