A standardized workflow to utilise a grid-computing system through advanced message queuing protocols. (October 2016)

• Record Type: Journal Article
• Authors: Zheng, Bangyou; Holland, Edward; Chapman, Scott C.
• Is Part Of: Environmental modelling & software. Volume 84(2016:Oct.)
• Page Start: 304
• Page End: 310
• Publication Date: 2016-10
• Subjects: HPC -- Condor -- Message queue -- Computer cluster -- Adaptation -- Crop breeding; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2016.07.012
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: Point-based crop models are frequently used to investigate the interaction of genotype, environment and management (G × E × M) for breeding programs and adaptation research. These studies require the processing of millions of simulations to make assessments in national scales. Here we demonstrate a platform to run crop simulations with HTCondor as implemented across more than 12 000 cores on CSIRO network. The workflow of HTCondor usages was built through a server-client structure with standardized messages (simulations), which were queued by message queue sever. The new workflow maximized the performance of the CSIRO HTCondor service and could simultaneously utilize more than 8000 cores. A case study of 4.8 million APSIM simulations (ca. 2 mins each on model desktop), was completed in 32.5 h. This platform makes it is possible to assess all combinations of G × E × M in a short period. The generic platform has since been used to run other massive serial processing tasks. Highlights: A standardized workflow implemented into the HTCondor cluster through a server-client structure to run any Windows programs. Advanced message queuing protocol as the centralized server to queue the standardized simulations. The new workflow to maximize the performance of CSIRO HTCondor service to simultaneously utilize more than 8000 cores. Platform is 50 times faster than the largest dedicated cluster within CSIRO, and provides economic benefits for organization. The platform has been used in several projects about climate adaptation, trait evaluation and statistical analysis.