An implementation of MPI and hybrid OpenMP/MPI parallelization strategies for an implicit 3D DDG solver. (15th June 2022)

• Record Type: Journal Article
• Authors: He, Xiaofeng; Wang, Kun; Feng, Yiwei; Lv, Lili; Liu, Tiegang
• Is Part Of: Computers & fluids. Volume 241(2022)
• Publication Date: 2022-06-15
• Subjects: Direct discontinuous Galerkin -- OpenMP parallelization -- MPI parallelization -- Hybrid openMP/MPI parallelization; Fluid dynamics -- Data processing -- Periodicals; 532.050285
• DOI: 10.1016/j.compfluid.2022.105455
• Languages: English
• ISSNs: 0045-7930

Abstract

Abstract: This work describes an implementation of OpenMP, MPI, hybrid OpenMP/MPI parallelization strategies for an implicit three-dimensional (3D) direct discontinuous Galerkin (DDG) solver used for Navier–Stokes equations. Significantly, an efficient local matrix-based MPI parallelization strategy of an implicit DG solver is proposed. Furthermore, we give an implementation of OpenMP and hybrid OpenMP/MPI strategy for comparison. The storage structure based on the local matrix makes the MPI parallelization can easily use the local number to access and assign, the storage is compact and compatible with Block Sparse Row (BSR) format, and the program is easier to modularize. Several numerical tests for 3D Navier–Stokes equations are implemented to indicate the performance of parallelization strategies. For the problem of more than 200 million degrees of freedom, the designed pure MPI strategy for 3rd-order DDG solvers with 2nd-order polynomials (DDG(P2)) can get parallel efficiency of almost 90% at near ten thousand cores on the Tianhe-2 supercomputer. In particular, the pure MPI parallelization based on local matrix reaches a higher level of parallel efficiency than hybrid OpenMP/MPI parallelization. Highlights: We give OpenMP, MPI, Hybrid OpenMP/MPI parallel strategies for 3D DDG solvers. We propose an local-matrix-based MPI strategy for an implicit DDG solver. The strategy holds compactness, high efficiency, convenience, and modularity. The pure MPI strategy gets the highest parallel efficiency in comparison. A MPI DDG(P2) solver gets a parallel efficiency of 89.5 at 9216 cores on Tianhe-2.