TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML. (21st October 2018)

• Record Type: Journal Article
• Authors: Kurth, Thorsten; Smorkalov, Mikhail; Mendygral, Peter; Sridharan, Srinivas; Mathuriya, Amrita
• Other Names: Thota Abhinav guestEditor.; He Yun guestEditor.; Hart David guestEditor.; Dahan Maytal guestEditor.; Navratil Paul guestEditor.
• Is Part Of: Concurrency and computation. Volume 31:Number 16(2019)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2018-10-21
• Subjects: deep learning -- performance -- scalability; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.4989
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Deep learning has proven to be a successful tool for solving a large variety of problems in various scientific fields and beyond. In recent years, the models as well as the available datasets have grown bigger and more complicated, and thus, an increasing amount of computing resources is required in order to train these models in a reasonable amount of time. Besides being able to use HPC resources, deep learning model developers want flexible frameworks which allow for rapid prototyping. One of the most important of these frameworks is Google TensorFlow, which provides both features, ie, good performance as well as flexibility. In this paper, we discuss different solutions for scaling the TensorFlow Framework to thousands of nodes on contemporary Cray XC supercomputing systems.