shmem4py: High-Performance One-Sided Communication for Python Applications

Marcin Rogowski, Jeff R. Hammond, David E. Keyes, Lisandro Dalcin

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

This paper describes shmem4py, a Python wrapper for the OpenSHMEM application programming interface (API) which follows a design similar to that of the well-known mpi4py package. OpenSHMEM is a descendant of the one-sided communication library for the Cray T3D and it is known for its uncompromising performance for low-latency and high-throughput use cases involving one-sided and collective communication. OpenSHMEM is arguably one of the most efficient and portable abstractions for modern network architectures. Thanks to tight interoperability with NumPy, shmem4py provides a convenient parallel programming framework leveraging both the high-productivity NumPy feature set and the high-performance networking capabilities of OpenSHMEM. This paper discusses the design and performance characteristics of shmem4py in a variety of communication patterns relative to lower-level languages (C) as well as MPI and mpi4py.

Original languageEnglish (US)
Title of host publicationProceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
PublisherAssociation for Computing Machinery
Pages1185-1193
Number of pages9
ISBN (Electronic)9798400707858
DOIs
StatePublished - Nov 12 2023
Event2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 - Denver, United States
Duration: Nov 12 2023Nov 17 2023

Publication series

NameACM International Conference Proceeding Series

Conference

Conference2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Country/TerritoryUnited States
CityDenver
Period11/12/2311/17/23

Keywords

  • High Performance Computing
  • MPI
  • OpenSHMEM
  • Python
  • shared memory

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Fingerprint

Dive into the research topics of 'shmem4py: High-Performance One-Sided Communication for Python Applications'. Together they form a unique fingerprint.

Cite this