HPE Cray MPI 9.2.0

Purpose

HPE Cray MPI 9.2.0 is a version of HPE Cray MPI that supports the MPI 4.1 standard. HPE Cray MPI 9.2.0 is based upon ANL MPICH 4.3.0 with support for libfabric and is optimized for Slingshot networks. HPE Cray MPI 9.2.0 is built to work with libfabric ABI version 1.6 or later (libfabric API version 1.14 or later).

HPE Cray MPI 9.x is the recommended production version of HPE Cray MPI beginning with CPE 25.09.

New Features in this Release

NEW CRAY MPI FEATURES

The following are new features in HPE Cray MPI.

New HPE Supercomputing Programming Software (SUPS) Delivery Model

Starting with the 26.09 release, HPE has transitioned from delivering a single, monolithic CPE release to a containerized SUPS release. The SUPS container provides a curated collection of HPE-proprietary, vendor-supplied and open-source tools tailored for HPC customers. HPE Cray MPI is a core component of the SUPS release.

To align with this new delivery model, HPE Cray MPI introduces enhancements that improve flexibility, strengthen standards adherence, and simplify operations within the SUPS ecosystem. These changes provide a more modern user experience while ensuring tight integration with other SUPS components.

Removal of MPI Library Name-Mangling

To align with standard MPI implementations, HPE Cray MPI 9.2.0 has adopted conventional library naming. The MPI libraries are now named libmpi.so and libmpifort.so. In prior releases, library names included compiler-specific suffixes (for example, libmpi_amd.so or libmpi_cray.so). Multiple compiler toolchains remain fully supported, each within its own installation directory.

This change eliminates the need for a dedicated MPICH ABI component directory, as the standardized library names are now consistent with other MPICH ABI-compatible implementations.

Changes to Compiler Wrapper Usage

SUPS containers do not provide the legacy CPE programming environment. As a result, the CrayPE compiler wrapper scripts (cc, CC, and ftn) and the associated CPE module environments are no longer available for compiling and linking MPI applications.

Starting with the SUPS 26.09 release, the HPE Cray MPI wrapper scripts mpicc, mpifort, and mpicxx are the primary and recommended interfaces to use for building MPI programs. The wrapper scripts have been enhanced with additional options:

   -help                - Displays HPE Cray MPI wrapper-specific options and flags
   -show                - Print the compile and link commands without executing them
   -pmi=<choice>        - PMI library to link: craypmi (default), pmi2, pmix, none
   -gpu=<choice>        - GPU transport layer: none (default), amd, nvidia
   -rpath               - Embed RPATH for all library paths added by this wrapper
   -no-rpath            - Omit RPATH for wrapper-added libraries (normal loader rules)
   -no-auto-libfabric   - Disable automatic libfabric detection and linking
   -no-auto-hugetlbfs   - Disable automatic hugepages detection and linking
   -no-auto-xpmem       - Disable automatic XPMEM detection and linking
   -no-auto-cray-opts   - Disable all Cray auto-detection (libfabric, hugepages, xpmem)

Refer to the mpicc(1) man page for detailed information on wrapper usage and options. Use of the -rpath option is recommended to ensure consistent and predictable library loading at runtime.

Changes to cray-mpich Module Files

SUPS containers do not provide module system integration by default. However, sites may choose to deploy a module system (such as Lmod or TCL) within custom containers or on host systems.

To support such configurations, HPE Cray MPI 9.2.0 provides simplified module file templates. These modules configure PATH, MANPATH, and LD_LIBRARY_PATH to ensure that the correct compiler-specific binaries and libraries are located at both build and run time. A distinct cray-mpich module file is provided for each supported compiler variant. These templates are located in a modulefiles directory beneath each compiler-specific HPE Cray MPICH installation path. Sites may customize these module templates as needed.

PMIx Support

Support for PMIx (Process Management Interface for Exascale) has been added in HPE Cray MPI 9.2.0. Cray MPI now supports three PMI implementations:

 Cray PMI (default) 
 PMIx
 PMI2 (e.g. slurm's PMI2 library)

This provides increased flexibility in the launch process. By default, the mpicc and mpifort compiler wrappers link against Cray PMI. To use PMIx, specify -pmi=pmix when compiling with the mpicc/mpifort wrappers, and ensure that the appropriate PMIx library path is available at link and run time. When launching Cray MPI applications with PMIx, the selected launcher (e.g., srun or mpiexec) must also support and be configured to use PMIx.

Dynamic Linking of Support Libraries

Starting with version 9.2.0, HPE Cray MPI supports dynamic runtime detection of support libraries, including XPMEM, the GPU Transport Layer (GTL), and multiple PMI implementations, improving flexibility across diverse build and execution environments. When these libraries are not explicitly linked at compile time, MPI can automatically discover and load them at runtime using dlopen(). This allows applications, such as precompiled binaries, to take advantage of features like enhanced intra-node performance with XPMEM, GPU support with GTL, and broader PMI compatibility (CrayPMI, PMI2, or PMIx) without requiring rebuilds. For optimal launch performance, explicit linking of these libraries at compile time is still recommended. However, dynamic runtime detection provides a flexible alternative when standard linking is not practical. For more details, see the mpicc man page and new environment variables: MPICH_USE_DLOPEN, MPICH_DLOPEN_DEBUG and MPICH_JOB_PMI.

Runtime Detection of PMI Implementation

If a PMI implementation is selected at link time using the mpicc or mpifort compiler wrappers, that implementation will be used at runtime. If no PMI library is linked into the application (e.g. -pmi=none), starting with 9.2.0, Cray MPI will automatically detect the PMI implementation used by the launcher and dynamically load the appropriate PMI library at runtime. This enables a single application binary to be launched with different PMI implementations without requiring recompilation. For additional details on dynamic PMI runtime detection, see the environment variables MPICH_USE_DLOPEN, MPICH_DLOPEN_DEBUG and MPICH_JOB_PMI.

Renaming of Cray PMI Libraries

As part of the PMI runtime detection feature, starting with Cray PMI 6.1.18, the PMI libraries have been renamed to include a unique “_cray” suffix. The new library names are libpmi2_cray.so and libpmi_cray.so. For backward compatibility, symlinks to the previous names (libpmi2.so and libpmi.so) are also provided. For more information, see the section LIBRARY NAMES AND COMPATIBILITY SYMLINKS in the intro_pmi man page.

MPICH_GPU_SUPPORT_ENABLED Behavior Changes

With Cray MPI 9.2.0, the default behavior of the MPICH_GPU_SUPPORT_ENABLED environment variable has changed. During MPI_Init, Cray MPI now detects whether the application is linked with the GPU Transport Layer (GTL) library. If GTL is present, GPU support is automatically enabled by internally setting MPICH_GPU_SUPPORT_ENABLED=1. This behavior provides a safety mechanism when the variable is not explicitly set, helping to avoid runtime errors such as segmentation faults or cxil_map write failures in libfabric when GPU memory buffers are used.

MPICH_GPU_SUPPORT_ENABLED may be set to 1, 0, or left unset. The default is unset.

  1 - enables GPU support. GPU buffers may be passed to MPI communication routines.
  0 - disables GPU support. Only CPU buffers may be used in MPI communication routines.
  unset - if the GTL library is linked in, GPU support is automatically enabled. 
          If GTL is not linked in, GPU support is automatically disabled.

Applications that link with GTL but only use CPU buffers in MPI operations, are encouraged to explicitly set MPICH_GPU_SUPPORT_ENABLED to 0 to avoid the overhead and resources associated with automatically enabling GPU support in both MPICH and libfabric.

Incompatible ROCM IPC Mechanism

ROCm versions 7.1.0 and 7.2.0 through 7.14 enable a new IPC mechanism which is incompatible with the GTL library, resulting in multiple issues including program aborts. Until the issues are resolved in a future ROCm version, it is highly recommended to enable the legacy ROCm IPC mechanism by setting the ROCm environment variable HSA_ENABLE_IPC_MODE_LEGACY=1 prior to program launch. Cray MPI will detect and warn users about this condition. For more details, see the Cray MPI environment variable GTL_ENABLE_ROCM_LEGACY_WARNING.

New Fortran Compiler Support

Cray MPI 9.2.0 introduces support for the AMD Fortran compiler (amdflang) provided with ROCm 7.13. The MPI Fortran 2008 (MPI_F08) language bindings are fully supported with this compiler.

In addition, Cray MPI 9.2.0 provides beta support for the MPI_F08 bindings with the NVIDIA 26.3 Fortran compiler. This support is preliminary, and some known issues remain. Relevant bug reports have been filed with NVIDIA.

Support for Multiple NICs per Process

Cray MPI 9.2.0 introduces support for multiple NICs per MPI rank. This feature enables a single MPI rank to transmit and receive data across multiple network interfaces using message hashing and/or large-message striping. The primary use case is systems where the number of NICs per node exceeds the number of MPI ranks per node. Leveraging multiple NICs per rank can improve performance by utilizing the aggregate network bandwidth available on the node. To configure and control this feature, refer to the following new environment variables: MPICH_CH4_OFI_ENABLE_MULTI_NIC_HASHING, MPICH_CH4_OFI_ENABLE_MULTI_NIC_STRIPING, MPICH_CH4_OFI_MAX_NICS, MPICH_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLD, MPICH_OFI_NIC_STATS, MPICH_NO_WILDCARDS and MPICH_OFI_LRANK_MAPPING.

Removal of craype-hugepages Modules

Starting with SUPS, the craype-hugepages modules are no longer provided. Despite this change, running MPI applications with huge pages remains recommended to improve performance. For more details see the “Running with Huge Pages” section in the intro_mpi man page. For best performance, users are encouraged to enable malloc allocations to be backed by 2 MB huge pages by setting “GLIBC_TUNABLES=glibc.malloc.hugetlb=2097152”

New MPI_Gather and MPI_Scatter Algorithms

Cray MPI 9.2.0 introduces new optimized MPI_Gather and MPI_Scatter algorithms. These algorithms are enabled by default for selected collective operations based on message size and communicators. For more details, see the new Cray MPI environment variables MPICH_GATHER_MIN_MSG_SIZE, MPICH_GATHER_MAX_COMM_SIZE, MPICH_GATHER_SYNC_FREQ, MPICH_SCATTER_MIN_MSG_SIZE, MPICH_SCATTER_MAX_COMM_SIZE and MPICH_SCATTER_SYNC_FREQ.

New MPI_Alltoall Algorithm

Cray MPI 9.2.0 introduces a new MPI_Alltoall algorithm based on one-sided transfer protocols. This algorithm can improve performance for small to medium-sized messages, particularly when multiple ranks share a network interface and at large process counts. For more details on enabling this algorithm, see the Cray MPI environment variable MPICH_ALLTOALL_ONESIDED_ALGORITHM.

Additional Release Notes

  • Starting with the 9.1.0 release, details of collective algorithm selection have been documented in /opt/cray/pe/mpich//share/

    • README.coll.alg_sel explains the procedure and provides an example.

    • README.coll.envvar documents environment variables for collective algorithm selection.

Product and OS Dependencies

The HPE Cray MPI 9.2.0 release is supported on the following HPE systems:

  • HPE Cray EX systems

On systems with AMD GPUs, HPE Cray MPI 9.2.0 supports ROCm 7.13. On systems with Nvidia GPUs, HPE Cray MPI 9.2.0 supports Cuda 13. On Slingshot systems, HPE Cray MPI uses the libfabric “cxi” provider by default.

Product Dependencies: cray-pals >= 1.0.6 cray-pmi >= 6.1.18 libfabric >= 1.9.0

One or more compilers:

  • AMD ROCM 7.13 or later

  • AOCC 5.0 or later

  • CCE 20.0.0 or later

  • GNU 12.3.0 or later

  • Intel 2022.1.0 or later

  • Nvidia 26.3 or later

Notes and Limitations

Limitations in HPE Cray MPI 9.2.0:

- HPE Cray MPI 9.2.0 does not support C++ language bindings.  Note
  that HPE Cray MPI 8.x releases also does not support C++ language
  bindings.  There is no plan to add C++ language bindings at this time.

- GPU Async Communication - Stream Triggered and Kernel Triggered
  communication - is currently unsupported.  Both features are planned
  to be added in a future release.

- Support of SS-10 and InfiniBand has been dropped.

- On systems with Slingshot-11 hardware, some MPI programs using
  MPI_PROBE (or its variants) experience an application failure,
  from libfabric, with the error message

    libfabric:766248:1740758749::cxi:core:cxip_claim_onload_cb():2581<warn>
    RXC (0x173:0) PtlTE 16:[Fatal] Unexpected event SEARCH, rc = NO_MATCH

  This problem is being tracked by NETCASSINI-7129 and a solution is
  expected in a future release of Slingshot Host Software (SHS).

  In the interim, this problem may be avoided by setting

    FI_CXI_RX_MATCH_MODE=software

  in the software environment where the application is run. intro_mpi(1)
  has information about FI_CXI_RX_MATCH_MODE.

- ROCm versions 7.1.0 and 7.2.0 through 7.14 enable a new IPC mechanism
  that has been found to be incompatible with the GTL library resulting
  in multiple issues including program aborts. It is recommended that the
  legacy IPC mechanism in ROCm be enabled to avoid these problems using
  the environment setting 'HSA_ENABLE_IPC_MODE_LEGACY=1' till the issues
  are resolved in a future ROCm version. GTL library will output a warning
  for the affected versions in stderr.

- Segfaults with AMD GPUs have been observed when applications or libraries
  do a memory copy between the host and an AMD GPU, with AMD GPU driver
  version 6.19.4, and potentially others. This can be worked around in MPT
  by setting GTL_HSA_VSMSG_CUTOFF_SIZE to a value greater than the largest
  memory transfer performed by the application. This causes the GTL library
  to use an alternate copy mechanism that may avoid the issue, which may also
  reduce performance.

- HPE has chosen to provide Fortran 2008 MPI language bindings for NVIDIA
  compilers beginning with the NVIDIA 26.3 release. We have observed
  problems with these bindings that we believe are NVIDIA compiler
  issues. These issues have been reported to NVIDIA. Defining where the
  problems are is difficult other than to say that there are general
  array support issues. We have seen failures of test cases with very
  simple and very complex use of arrays but other very similar tests
  pass. Some of the failures are intermittent.

- The Slingshot Host Software CXI driver version 13.0 has a bug that
  prevents the use of MPI spawn on Slingshot fabrics. This bug was fixed in
  driver version 14.0.1.

Other Notes

- This version of HPE Cray MPI (9.2.0) supports CCE 21. 

- Some versions of NVIDIA compilers, in particular NVIDIA 25.3 and 25.5
  compilers, add "/usr/lib64" to the rpath of any program linked with
  the NVIDIA compiler drivers.  This rpath blocks the use of other
  parts of Cray PE.  In particular it prevents Cray PMI from being used.
  The most common run-time error one sees if using Slurm PMI with HPE
  Cray MPI when Cray PMI is expected is:

  ../src/mpid/common/cray/cray_pmi_utils.c:108: _mpidi_cray_pmi_send: Assertion `rc == PMI2_SUCCESS' failed

  One way to avoid this behavior is to add the NVIDA compiler option
  "-Mnorpath" when linking programs with NVIDIA compilers.

Documentation

For more information see the intro_mpi man page.

License

Except for the third party components and software licensed by HPE through proprietary agreements, components, files or programs contained within this package or product are Copyright -2026 Hewlett Packard Enterprise Development LP.

Attribution notices for open source licensed software for this package are detailed in the file: /opt/cray/pe/mpich/9.2.0/ATTRIBUTIONS Copyright -2026 Hewlett Packard Enterprise Development LP