TY - JOUR
T1 - The PetscSF Scalable Communication Layer
AU - Zhang, Junchao
AU - Brown, Jed
AU - Balay, Satish
AU - Faibussowitsch, Jacob
AU - Knepley, Matthew
AU - Marin, Oana
AU - Mills, Richard Tran
AU - Munson, Todd
AU - Smith, Barry F.
AU - Zampini, Stefano
N1 - KAUST Repository Item: Exported on 2021-11-21
Acknowledgements: We thank Akhil Langer and Jim Dinan from the NVIDIA NVSHMEM team for their assistance. This work was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration, and by the U.S. Department of Energy under Contract DE-AC02-06CH11357 and Office of Science Awards DESC0016140 and DE-AC02-0000011838. This research used resources of the Oak Ridge Leadership Computing Facilities, a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.
PY - 2021
Y1 - 2021
N2 - PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETScs communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. In this paper, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.
AB - PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETScs communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. In this paper, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.
UR - http://hdl.handle.net/10754/667821
UR - https://ieeexplore.ieee.org/document/9442258/
UR - http://www.scopus.com/inward/record.url?scp=85107218066&partnerID=8YFLogxK
U2 - 10.1109/TPDS.2021.3084070
DO - 10.1109/TPDS.2021.3084070
M3 - Article
SN - 2161-9883
SP - 1
EP - 1
JO - IEEE Transactions on Parallel and Distributed Systems
JF - IEEE Transactions on Parallel and Distributed Systems
ER -