Operating two InfiniBand grid clusters over 28 km distance. (10th October 2011)

• Record Type: Journal Article
• Authors: Richling, Sabine; Hau, Steffen; Kredel, Heinz; Kruse, Hans-Günther
• Is Part Of: International journal of grid and utility computing. Volume 2:Number 4(2011)
• Page Start: 303
• Page End: 312
• Publication Date: 2011-10-10
• Subjects: operating clusters -- long-distance InfiniBand -- performance model -- grid utility -- computing; Electronic data processing -- Distributed processing -- Periodicals; Electronic commerce -- Management -- Computer programs -- Periodicals; 004.605
• Languages: English
• ISSNs: 1741-847X

Abstract

This paper considers an InfiniBand connection between two bwGRiD clusters over a distance of 28 km in day-to-day production use. We discuss the hardware set-up of InfiniBand messages converted and transported over a fibre optic connection. The two clusters can be operated as single system image, the batch system will enforce that all nodes for a job are allocated on one side of the cluster. This is to optimise MPI performance, which would not be sufficient for communication between nodes on opposite sides of the 28 km connection. We report on the successful solution of all technical and organisational integration hurdles. By a simple performance model we discuss the anticipated costs for a doubling in communication performance.