Use of the ERD for administrative monitoring of Theta. (29th January 2019)

• Record Type: Journal Article
• Authors: Kristiansen, Alex
• Other Names: Thota Abhinav guestEditor.; He Yun guestEditor.; Hart David guestEditor.; Dahan Maytal guestEditor.; Navratil Paul guestEditor.
• Is Part Of: Concurrency and computation. Volume 31:Number 16(2019)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2019-01-29
• Subjects: Cray -- ERD -- Golang -- HPC -- monitoring -- Theta -- XC40; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5099
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Monitoring the state of an HPC cluster in a timely and accurate fashion is critical to most system administration functions. For many Cray users, the first step in monitoring is ingestion of log files. Unfortunately, log parsing is an inherently inefficient process, requiring multiple software components to read and write from files on disk. Cray's own utilities use a message bus, the Event Router Daemon (ERD), for a wide variety of purposes. At the Argonne Leadership Computing Facility (ALCF), we have begun to use this message bus for monitoring via a client library written in Go, allowing us to read in structured data directly from Cray's services and, in many instances, bypass log files entirely. In this paper, we will examine the implementation and utilization of this approach on our 4392 node XC40, Theta, as well as the overall benefits and drawbacks to using the ERD for real‐time monitoring.