OpenACC Acceleration of Parallel Direct Numerical Simulation of Turbulent Flows

Document Type : Research Paper

Authors
Ahmed Abdellatif
Joan Grau
Ricardo Torres
Lluís Jofre
Department of Fluid Mechanics, Universitat Politècnica de Catalunya · BarcelonaTech (UPC), Barcelona 08034, Spain
10.22055/jacm.2024.47026.4646
Abstract
This work presents a state-of-the-art computational approach for performing large-scale simulations of turbulence on heterogeneous compute nodes. The approach proposed combines message passing interface (MPI) parallelization for distributing the computation across nodes with open accelerators (OpenACC) to target a graphical processing unit (GPU) acceleration. Two different OpenACC approaches, corresponding to managed and non-managed data management, have been tested and compared. The results obtained indicate that the non-managed strategy outperforms the non-managed scheme in all cases. Additionally, the non-managed GPU-accelerated approach presents an overall speedup of 6× with respect to the CPU version. Finally, the capability of the resulting parallel and accelerated flow solver to perform high-fidelity simulations of turbulent flow is validated against a canonical wall-bounded fluid problem.
Keywords
Compressible flow
Computational fluid dynamics
MPI
OpenACC
Subjects

Publisher’s Note Shahid Chamran University of Ahvaz remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

[1] Masquelet, M., Yan, J., Dord, A., Laskowski, G., Shunn, L., Jofre, L., et al., Uncertainty Quantification in Large Eddy Simulations of a Rich-Dome Aviation Gas Turbine, ASME Turbo Expo 2017: Turbomachinery Technical Conference and Exposition, 2017.
[2] Jofre, L., Urzay, J., Transcritical diffuse-interface hydrodynamics of propellants in high-pressure combustors of chemical propulsion systems, Progress in Energy and Combustion Science, 82, 2021, 100877.
[3] Scarano, F., Overview of PIV in Supersonic Flows, Springer Berlin Heidelberg, 2008.
[4] Alfonsi, G., Reynolds-Averaged Navier–Stokes Equations for Turbulence Modeling, Applied Mechanics Reviews, 62, 2009, 040802.
[5] Larsson, J., Wang, Q., The prospect of using large eddy and detached eddy simulations in engineering design, and the research required to get there, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 372, 2014, 20130329.
[6] Moin, P., Mahesh, K., Direct Numerical Simulation: A Tool in Turbulence Research, Annual Review of Fluid Mechanics, 30, 1998, 539-578.
[7] Alowayyed, S., Groen, D., Coveney, P.V., Hoekstra, A.G., Multiscale computing in the exascale era, Journal of Computational Science, 22, 2017, 15-25.
[8] Salvadore, F., Bernardini, M., Botti, M., GPU accelerated flow solver for direct numerical simulation of turbulent flows, Journal of Computational Physics, 235, 2013, 129-142.
[9] Witherden, F.D., Farrington, A.M., Vincent, P.E., PyFR: An open source framework for solving advection–diffusion type problems on streaming architectures using the flux reconstruction approach, Computer Physics Communications, 185, 2014, 3028-3040.
[10] Shi, X., Agrawal, T., Lin, C.-A., Hwang, F.-N., Chiu, T.-H., A parallel nonlinear multigrid solver for unsteady incompressible flow simulation on multi-GPU cluster, Journal of Computational Physics, 414, 2020, 109447.
[11] Zhang, J.-L., Chen, H.-Q., Xu, S.-G., Gao, H.-Q., A novel GPU-parallelized meshless method for solving compressible turbulent flows, Computers & Mathematics with Applications, 80, 2020, 2738-2763.
[12] Dubois, R., Goncalves da Silva, E., Parnaudeau, P., High performance computing of stiff bubble collapse on CPU-GPU heterogeneous platform, Computers & Mathematics with Applications, 99, 2021, 246-256.
[13] Costa, P., A FFT-based finite-difference solver for massively-parallel direct numerical simulations of turbulent flows, Computers & Mathematics with Applications, 76, 2018, 1853-1862.
[14] Bernardini, M., Modesti, D., Salvadore, F., Pirozzoli, S., STREAmS: A high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows, Computer Physics Communications, 263, 2021, 107906.
[15] Merzari, E., Leite, V., Fang, J., Shaver, D., Min, M., Kerkemeier, S., et al., Energy Exascale CFD Simulations with the Spectral Element Method, Journal of Fluids Engineering, 146, 2024, 1-57.
[16] Offermans, N., Marin, O., Schanen, M., Gong, J., Fischer, P., Schlatter, P., et al., On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems, Proceedings of the Exascale Applications and Software Conference 2016, Stockholm, Sweden, 2016.
[17] Swirydowicz, K., Chalmers, N., Karakus, A., Warburton, T., Acceleration of tensor-product operations for high-order finite element methods, The International Journal of High Performance Computing Applications, 33, 2019, 735-757.
[18] Medina, D., OKL: A unified language for parallel architectures, Ph.D. Thesis, Rice University, USA, 2015.
[19] Oak Ridge National Laboratory:  OLCF Summit, https://www.olcf.ornl.gov/olcf-resources/compute-systems/summit, 2020.
[20] Karp, M., Massaro, D., Jansson, N., Hart, A., Wahlgren, J., Schlatter, P., et al., Large-Scale direct numerical simulations of turbulence using GPUs and modern Fortran, The International Journal of High Performance Computing Applications, 37, 2023, 487-502.
[21] Jansson, N., Karp, M., Podobas, A., Markidis, S., Schlatter, P., Neko: A modern, portable, and scalable framework for high-fidelity computational fluid dynamics, Computers & Fluids, 275, 2024, 106243.
[22] Massaro, D., Karp, M., Jansson, N., Markidis, S., Schlatter, P., Direct numerical simulation of the turbulent flow around a Flettner rotor, Scientific Reports, 14, 2024, 3004.
[23] Message Passing Interface Forum: MPI: A Message-Passing Interface Standard Version 3.1, https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf, 2015.
[24] OpenACC-Standard.org: The OpenACC Application Programming Interface 2.6, https://www.openacc.org/sites/default/files/inline-files/OpenACC.2.6.final.pdf, 2017.
[25] Jofre, L., Abdellatif, A., Oyarzun, G., RHEA: an open-source Reproducible Hybrid-architecture flow solver Engineered for Academia, Journal of Open Source Software, 8, 2023, 4637.
[26] Jofre, L., del Rosario, Z.R., Iaccarino, G., Data-driven dimensional analysis of heat transfer in irradiated particle-laden turbulent flow, International Journal of Multiphase Flow, 125, 2020, 103198.
[27] Sutherland, W., LII. The viscosity of gases and molecular force, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 36, 1893, 507-531.
[28] Linstrom, P., Mallard, W., NIST chemistry webbook-srd 69, National Institute of Standards and Technology, Gaithersburg, MD, 2001.
[29] Gottlieb, S., Shu, C.-W., Tadmor, E., Strong stability-preserving high-order time discretization methods, SIAM Review, 43, 2001, 89-112.
[30] Abdellatif, A., Ventosa-Molina, J., Grau, J., Torres, R., Jofre, L., Artificial compressibility method for high-pressure transcritical fluids at low Mach numbers, Computers & Fluids, 270, 2024, 106163.
[31] Coppola, G., Capuano, F., Pirozzoli, S., de Luca, L., Numerically stable formulations of convective terms for turbulent compressible flows, Journal of Computational Physics, 382, 2019, 86-104.
[32] Coppola, G., Capuano, F., de Luca, L., Discrete energy-conservation properties in the numerical simulation of the Navier–Stokes equations, Applied Mechanics Reviews, 71, 2019, 010803.
[33] Frontier, ORNL’s Exascale supercomputer designed to deliver world-leading performance in 2021, 2022.
[34] Evans, T.M., Siegel, A., Draeger, E.W., Deslippe, J., Francois, M.M., Germann, T.C., et al., A survey of software implementations used by application codes in the Exascale Computing Project, The International Journal of High Performance Computing Applications, 36, 2022, 5-12.
[35] Khronos, SYCL is a royalty-free, cross-platform abstraction layer enables code for heterogeneous and offload processors to be written using modern ISO C++, 2018.
[36] Carter Edwards, H., Trott, C.R., Sunderland, D., Kokkos: Enabling manycore performance portability through polymorphic memory access patterns, Journal of Parallel and Distributed Computing, 74, 2014, 3202-3216.
[37] NVIDIA Nsight, https://docs.nvidia.com/nsight-systems/UserGuide/index.html#profiling-from-the-gui, 2020.
[38] Nielsen, F., Introduction to HPC with MPI for Data Science, Springer, 2016.
[39] Barcelona Supercomputing Center, BSC-CNS | Centro Nacional de Supercomputación, https://www.bsc.es, 2020.
[40] NVIDIA Tools Extension Library (NVTX), https://docs.nvidia.com/nsight-systems/UserGuide/index.html#nvtx-trace
[41] Smits, A.J., McKeon, B.J., Marusic, I., High–Reynolds number wall turbulence, Annual Review of Fluid Mechanics, 43, 2011, 353-375.
[42] Nelson, K., Fringer, O., Reducing spin-up time for simulations of turbulent channel flow, Physics of Fluids, 29, 2017, 105101.
[43] Appelhans, D., Auerbach, G., Averill, D., Black, R., Brown, A., Buono, D., et al., Functionality and performance of nvlink with ibm power9 processors, IBM Journal of Research and Development, 62, 2018, 1-10.
[44] Vincent, J., Gong, J., Karp, M., Peplinski, A., Jansson, N., Podobas, A., et al., Strong scaling of OpenACC enabled Nek5000 on several GPU based HPC systems, International Conference on High Performance Computing in Asia-Pacific Region, 94-102, 2022.
[45] Moser, R.D., Kim, J., Mansour, N.N., Direct numerical simulation of turbulent channel flow up to Reτ= 590, Physics of Fluids, 11, 1999, 943-945.
Journal of Applied and Computational Mechanics
Volume 11, Issue 2
Spring 2025
Pages 425-438