Hybrid parallelization of Euler–Lagrange simulations based on MPI-3 shared memory. (December 2022)

• Record Type: Journal Article
• Authors: Kopper, Patrick; Copplestone, Stephen M.; Pfeiffer, Marcel; Koch, Christian; Fasoulas, Stefanos; Beck, Andrea
• Is Part Of: Advances in engineering software. Volume 174(2022)
• Publication Date: 2022-12
• Subjects: High-performance computing -- Hybrid parallel programming -- Shared memory -- Particle-In-Cell -- Discontinuous Galerkin spectral element -- Halo region; Computer-aided engineering -- Periodicals; Engineering -- Computer programs -- Periodicals; Engineering -- Software -- Periodicals; Periodicals; 620.0028553
• DOI: 10.1016/j.advengsoft.2022.103291
• Languages: English
• ISSNs: 0965-9978

Abstract

Abstract: The use of Euler–Lagrange methods on unstructured grids extends their application area to more versatile setups. However, the lack of a regular topology limits the scalability of distributed parallel methods, especially for routines that perform a physical search in space. One of the most prominent slowdowns is the search for halo elements in physical space for the purpose of runtime communication avoidance. In this work, we present a new communication-free halo element search algorithm utilizing the MPI-3 shared memory model. This novel method eliminates the severe performance bottleneck of many-to-many communication during initialization compared to the distributed parallelization approach and extends the possible applications beyond those achievable with the previous approach. Building on these data structures, we then present methods for efficient particle emission, scalable deposition schemes for particle–field coupling, and latency hiding approaches. The scaling performance of the proposed algorithms is validated through plasma dynamics simulations of an open-source framework on a massively parallel system, demonstrating an efficiency of up to 80 % on 131 072 cores. Highlights: A novel method to identify halo elements for unstructured Euler–Lagrange solvers. Avoidance of many-to-many communication through use of MPI-3 shared memory. Extension to emission, latency hiding, and runtime deposition mechanisms. Implementation evaluated through scaling tests on a state-of-the-art supercomputer. Good initialization times and efficiency for both weak and strong scaling.