A distributed load balancing algorithm for climate big data processing over a multi‐core CPU cluster. (24th March 2016)

• Record Type: Journal Article
• Authors: Wang, Yuzhu; Jiang, Jinrong; Ye, Huang; He, Juanxiong
• Other Names: Bellatreche Ladjel guestEditor.; Mohania Mukesh guestEditor.; Luo Xiangfeng guestEditor.; Liu Yunhuai guestEditor.; Li Qing guestEditor.
• Is Part Of: Concurrency and computation. Volume 28:Number 15(2016)
• Page Start: 4144
• Page End: 4160
• Publication Date: 2016-03-24
• Subjects: big data -- data processing -- load imbalance -- optimization algorithm -- high performance computing; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3822
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Load imbalance is a common problem to be tackled urgently in large scale data‐driven simulation systems or data intensive computing. According to the coupler, the Chinese Academy of Sciences‐Earth System Model (CAS‐ESM) implements one‐way nesting of the Institute of Atmospheric Physics of Chinese Academy of Sciences Atmospheric General Circulation Model version 4.0 (IAP AGCM4.0) and Weather Research and Forecasting model (WRF). The METGRID (meteorological grid) and REAL program modules in the WRF are used to process meteorological data. In the CAS‐ESM, the load of the METGRID module is seriously unbalanced on many CPU cores. The load imbalance has a serious impact on the processing speed of meteorological data, so this study designs an optimization algorithm to solve the problem. Numerical experiments show that compared to before optimization, the optimization algorithm can solve the load imbalance of the METGRID, and the computation speed of the METGRID and REAL modules after optimization on 64 CPU cores is about 7.2 times faster than before. Meanwhile, the whole computation speed of the CAS‐ESM can improve by 217.53%. In addition, results indicate that they also can reach to a similar speedup on different numbers of CPU cores. Copyright © 2016 John Wiley & Sons, Ltd.