Heterogeneous computing on mixed unstructured grids with PyFR. (5th October 2015)

• Record Type: Journal Article
• Authors: Witherden, F.D.; Vermeire, B.C.; Vincent, P.E.
• Is Part Of: Computers & fluids. Volume 120(2015)
• Page Start: 173
• Page End: 186
• Publication Date: 2015-10-05
• Subjects: High methods -- Flux reconstruction -- Heterogeneous computing -- Cylinder flow; Fluid dynamics -- Data processing -- Periodicals; 532.050285
• DOI: 10.1016/j.compfluid.2015.07.016
• Languages: English
• ISSNs: 0045-7930

Abstract

Highlights: Perform high-order accurate unsteady simulations of flow over a cylinder with PyFR. We report on both the performance and the resulting turbulent statistics. Note the ability to capture both the H and the L wake modes at Re = 3900. Demonstrate novel framework that allows parallelisation on mixed CPU/GPU systems. Demonstrate the lack of performance portability associated with OpenCL. Abstract: PyFR is an open-source high-order accurate computational fluid dynamics solver for unstructured grids. In this paper we detail how PyFR has been extended to run on mixed element meshes, and a range of hardware platforms, including heterogeneous multi-node systems. Performance of our implementation is benchmarked using pure hexahedral and mixed prismatic-tetrahedral meshes of the void space around a circular cylinder. Specifically, for each mesh performance is assessed at various orders of accuracy on three different hardware platforms; an NVIDIA Tesla K40c GPU, an Intel Xeon E5-2697 v2 CPU, and an AMD FirePro W9100 GPU. Performance is then assessed on a heterogeneous multi-node system constructed from a mix of the aforementioned hardware. Results demonstrate that PyFR achieves performance portability across various hardware platforms. In particular, the ability of PyFR to target individual platforms with their 'native' language leads to significantly enhanced performance cf. targeting each platform with OpenCL alone. PyFR is also found to be performant on the heterogeneous multi-node system, achieving a significant fraction of the available FLOP/s. Finally, each mesh is used to undertake nominally fifth-order accurate long-time simulations of unsteady flow over a circular cylinder at a Reynolds number of 3900 using a cluster of NVIDIA K20c GPUs. Long-time dynamics of the wake are studied in detail, and results are found to be in excellent agreement with previous experimental/numerical data. All results were obtained with PyFR v0.2.2, which is freely available under a 3-Clause New Style BSD license (seewww.pyfr.org ).