HPE Slingshot SHMEM

[!important] HPE Slingshot SHMEM is not supported in the SUPS 1.0.0 release. All SHMEM related functionalities are disabled in this version of SUPS.

For detailed documentation about the product, refer to the HPE Slingshot SHMEM Man Pages.

HPE Slingshot SHMEM is installed inside the container only. There is nothing to install on the host system.

Installation

HPE Slingshot SHMEM is installed by default in the SUPS reference container images. The HPE Slingshot SHMEM package cray-shmem is provided by the Slingshot SHMEM RPM repository that is enabled by default in the SUPS reference container images.

To install HPE Slingshot SHMEM in a custom container image, follow the run-time container template to enable all necessary repositories and then install the cray-shmem package via the package manager:

# For Rocky Linux 9.6 development container
dnf install cray-pmi-devel cray-shmem

# For SLES 15 SP7 development container
zypper install cray-pmi-devel cray-shmem

HPE Slingshot SHMEM is installed at SHMEM_ROOT=/opt/cray/shmem/sma/12.0.0 in the container.

Compiling Applications

The SHMEM header files are located in ${SHMEM_ROOT}/ofi/include and the library files are located in ${SHMEM_ROOT}/ofi/lib64. The shared library name is libsma.so.

You also need to link in the Cray PMI libraries libpmi.so and libpmi2.so to run the containerized application on the cluster. These libraries are located in /opt/cray/pe/pmi/6.1.18/lib.

Here is a summary of the additional compiler and linker flags:

-I${SHMEM_ROOT}/ofi/include -L${SHMEM_ROOT}/ofi/lib64 -lsma -L/opt/cray/pe/pmi/6.1.18/lib -lpmi -lpmi2

Simply add these options to your existing compiler command line when compiling your SHMEM application in the container.

Running Applications

SUPS provides the hpc-container-wrapper utility to run containerized SHMEM application with HPE Slingshot SHMEM on HPE Slingshot systems. Please refer to the SUPS Utility Tools documentation for more details.

GPU support

To enable GPU support in your SHMEM application, you also need to link in the vendor-specific GPU Translation Libraries (GTL) provided by HPE Slingshot SHMEM.

For AMD GPU support, add the following linker flags

-L${SHMEM_ROOT}/lib -lsma_gtl_hsa

Similarly, for NVIDIA GPU support, add the following linker flags

-L${SHMEM_ROOT}/lib -lsma_gtl_cuda

Set the SHMEM_GPU_SUPPORT_ENABLED environment variable to 1 when running the application. This can be done by passing the --env SHMEM_GPU_SUPPORT_ENABLED=1 option to apptainer when launching the container.

Differences from CPE

If you are transitioning from a Cray Programming Environment (CPE) system, here are some of the differences when using HPE Slingshot SHMEM in the SUPS container environment.

With the CPE module environment, you can load the cray-shmem module to set up the SHMEM library paths and then use the CrayPE compiler wrappers (e.g. cc, CC, ftn) to compile SHMEM applications. The wrappers automatically include the correct header and library search paths and link in the correct libraries, including Cray PMI and libfabric dependencies. Optional GPU support can be enabled by loading the craype-accel-* module for the corresponding GPU architecture.

In the SUPS container environment, there is no module environment. You need to manually specify the compiler and linker flags to compile SHMEM applications, as described in the Compiling Applications section above. You are also responsible for linking in the Cray PMI libraries and, if needed, the GPU Translation Libraries (GTL) for GPU support.

Troubleshooting

Undefined reference to `fi_` symbols when linking with certain compilers

When linking your SHMEM application with HPE Slingshot SHMEM, you may encounter linker errors about undefined references to fi_ symbols. For example, with GCC 14, you might see errors like:

/opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: /opt/cray/shmem/sma/12.0.0/ofi/lib64/libsma.so: undefined reference to `fi_version@FABRIC_1.0'
/opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: /opt/cray/shmem/sma/12.0.0/ofi/lib64/libsma.so: undefined reference to `fi_dupinfo@FABRIC_1.3'
/opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: /opt/cray/shmem/sma/12.0.0/ofi/lib64/libsma.so: undefined reference to `fi_strerror@FABRIC_1.0'
/opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: /opt/cray/shmem/sma/12.0.0/ofi/lib64/libsma.so: undefined reference to `fi_freeinfo@FABRIC_1.3'
/opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: /opt/cray/shmem/sma/12.0.0/ofi/lib64/libsma.so: undefined reference to `fi_tostr@FABRIC_1.0'
/opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: /opt/cray/shmem/sma/12.0.0/ofi/lib64/libsma.so: undefined reference to `fi_fabric@FABRIC_1.1'
/opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: /opt/cray/shmem/sma/12.0.0/ofi/lib64/libsma.so: undefined reference to `fi_getinfo@FABRIC_1.3'
collect2: error: ld returned 1 exit status

This is because some linkers, such as the one used by GCC 14, is more strict about shared library dependencies.

There are several workarounds to resolve this issue:

rpath the libfabric library path by adding -Wl,-rpath,/opt/cray/libfabric/2.3.1/lib64 to the linker.
add -L/opt/cray/libfabric/2.3.1/lib64 -lfabric to the linker.
add -Wl,--allow-shlib-undefined to the linker to allow undefined symbols in shared libraries.