xthi User Tutorial

• Overview

• Prepare the Workspace

• Launch the Build Container

• Build the Application

• Create Run-Time Container Image

• Run the Containerized Application

  • Local Sanity Test (Login Node)

  • Slurm Workload Manager (Slurm)

    • Cray MPICH

    • Open MPI

  • PBS Professional (PBS Pro)

    • Cray MPICH

    • Open MPI

Overview

This user tutorial shows how to build and run a containerized MPI application with HPE Supercomputing Programming Software (SUPS).

The example in this guide is xthi, which is included with SUPS. xthi is a simple MPI “hello world” program. It prints a message from each OpenMP thread and MPI rank. It also prints the GPU devices that each MPI rank can access if GPU support is enabled at compile time.

Set the following shell variables before proceeding to the rest of the guide. Change the values to suit your environment. Some SUPS variables are set automatically by the sups-config utility tool. See SUPS Utility Tools - sups-config for more information.

eval "$(sups-config --get SUPS_VERSION)"
eval "$(sups-config --get SUPS_SOURCE_DIR)"
eval "$(sups-config --get SUPS_IMAGE_DIR)"
eval "$(sups-config --get SUPS_COMMON_DIR)"
STACK="amd" # or "nvidia"
DISTRO="rocky" # or "sles"
WORKSPACE="/path/to/workspace"

# For the Open MPI workflow only
OMPI_VERSION="6.0.0"
OMPI_COMPILER="nvidia"  # or "gnu"
OMPI_DIR="/opt/cray/sups/openmpi/${OMPI_VERSION}/${OMPI_COMPILER}"

Variable	Purpose
SUPS_VERSION	SUPS version, e.g., 1.0.0
SUPS_SOURCE_DIR	SUPS source directory
SUPS_IMAGE_DIR	SUPS container image directory
SUPS_COMMON_DIR	SUPS shared common directory with build artifacts and RPM repositories
STACK	SUPS software stack (amd or nvidia)
DISTRO	Target Linux distribution of the container images (rocky or sles)
WORKSPACE	Workspace path for this tutorial
OMPI_VERSION	SUPS Open MPI version. Required only for the Open MPI workflow.
OMPI_COMPILER	SUPS Open MPI compiler type. Required only for the Open MPI workflow.
OMPI_DIR	Open MPI install prefix. For the Open MPI workflow, use this in place of CRAY_MPICH_DIR and CRAY_PMI_DIR.

[!NOTE] This guide covers two MPI workflows. Cray MPICH uses hpc-container-wrapper --mpi=cray with Slurm --mpi=cray_shasta. SUPS-built Open MPI uses hpc-container-wrapper --mpi=ompi with Slurm --mpi=pmix. The wrapper supplies the required PMIx, GPU, and Slingshot integration for both workflows. The container build and runtime image steps are identical for both; only the make arguments and job launch settings differ.

Prepare the Workspace

Create the workspace directory if it does not already exist and change into that directory.

mkdir -p "${WORKSPACE}"
cd "${WORKSPACE}"

Copy the xthi source file and Makefile into that directory.

cp "${SUPS_SOURCE_DIR}/tools/xthi/Makefile" "${SUPS_SOURCE_DIR}/tools/xthi/xthi.c" "${WORKSPACE}"

Launch the Build Container

SUPS uses Apptainer as the container engine for building and running containerized applications.

The SUPS reference build image for Apptainer is prebuilt and stored in Singularity Image File (SIF) format at ${SUPS_IMAGE_DIR}/singularity/sups-${STACK}-${DISTRO}.sif.

Start a build container from the SIF image. Mount the workspace, set the container working directory to the mount point, and open an interactive shell.

apptainer shell --writable-tmpfs --fakeroot --no-home \
    --bind "${WORKSPACE}:/workspace:rw" --cwd /workspace \
    "${SUPS_IMAGE_DIR}/singularity/sups-${STACK}-${DISTRO}.sif"

[!IMPORTANT] Keep the session open and proceed to Build the Application to build the xthi application inside the Apptainer container.

Build the Application

A Makefile is provided for users to build the xthi application inside the build container. The application can be built for three different flavors: CPU-only, AMD GPU enabled, and NVIDIA GPU enabled. The following table summarizes the command-line arguments for the xthi Makefile.

Argument	Required	Description
GPU	No	GPU target of xthi. If not set, the build will default to CPU-only.
MPI	No	MPI implementation: cray (Cray MPICH, default) or ompi (SUPS Open MPI).
CC	No	C compiler command. Defaults to gcc.
CRAY_LIBFABRIC_DIR	Yes	Root path to Cray Libfabric installation.
CRAY_MPICH_DIR	Yes	Root path to Cray MPICH installation. For the Open MPI workflow, set this to ${OMPI_DIR}.
CRAY_PMI_DIR	Yes	Root path to Cray PMI installation. For the Open MPI workflow, set this to ${OMPI_DIR}.
ROCM_PATH	Conditional	Root path to ROCm installation. Required if GPU=AMD.
CUDA_PATH	Conditional	Root path to CUDA installation. Required if GPU=NVIDIA.

To build the CPU-only version, run make with the following command.

Here is an example make command for building the CPU-only version of xthi in the AMD Rocky Linux 9.6 reference build container.

make \
    CC=/opt/rh/gcc-toolset-14/root/usr/bin/gcc \
    CRAY_LIBFABRIC_DIR=/opt/cray/libfabric/2.3.1 \
    CRAY_MPICH_DIR=/opt/cray/pe/mpich/9.2.0/ofi/gnu/11.2 \
    CRAY_PMI_DIR=/opt/cray/pe/pmi/6.1.18

To build the CPU-only version of xthi in the AMD SLES 15 SP7 reference build container.

make \
    CC=/usr/bin/gcc-14 \
    CRAY_LIBFABRIC_DIR=/opt/cray/libfabric/2.3.1 \
    CRAY_MPICH_DIR=/opt/cray/pe/mpich/9.2.0/ofi/gnu/12.3 \
    CRAY_PMI_DIR=/opt/cray/pe/pmi/6.1.18

To enable AMD GPU support, add the following line to the end of the make command with the GPU and ROCM_PATH variables in an AMD build container.

GPU=AMD ROCM_PATH=/opt/rocm/core-7.13

Similarly, to enable NVIDIA GPU support, add the following line to the end of the make command with the GPU and CUDA_PATH variables in an NVIDIA build container.

GPU=NVIDIA CUDA_PATH=/opt/nvidia/hpc_sdk/Linux_aarch64/26.3/cuda/13.1

For the Open MPI workflow, the SUPS-built Open MPI is pre-installed in the Nvidia reference build container. Build xthi by setting MPI=ompi and pointing CRAY_MPICH_DIR and CRAY_PMI_DIR to the Open MPI installation prefix instead of the Cray MPICH paths. MPI=ompi links against libmpi only; the standalone Cray PMI libraries (-lpmi -lpmi2) are not linked because Open MPI provides PMIx inside libmpi.

Here is an example make command for building the CPU-only version of xthi with Open MPI in the Rocky Linux 9.6 reference build container.

OMPI_DIR="/opt/cray/sups/openmpi/6.0.0/nvidia"
make \
    MPI=ompi \
    CC=/opt/rh/gcc-toolset-14/root/usr/bin/gcc \
    CRAY_LIBFABRIC_DIR=/opt/cray/libfabric/2.3.1 \
    CRAY_MPICH_DIR="${OMPI_DIR}" \
    CRAY_PMI_DIR="${OMPI_DIR}"

To build the CPU-only version of xthi with Open MPI in the SLES 15 SP7 reference build container.

OMPI_DIR="/opt/cray/sups/openmpi/6.0.0/nvidia"
make \
    MPI=ompi \
    CC=/usr/bin/gcc-14 \
    CRAY_LIBFABRIC_DIR=/opt/cray/libfabric/2.3.1 \
    CRAY_MPICH_DIR="${OMPI_DIR}" \
    CRAY_PMI_DIR="${OMPI_DIR}"

The AMD and NVIDIA GPU flags apply in the same way as for Cray MPICH. For example, to build the NVIDIA GPU-enabled version of xthi with Open MPI in the SLES 15 SP7 reference build container, add the GPU and CUDA_PATH variables to the end of the make command.

OMPI_DIR="/opt/cray/sups/openmpi/6.0.0/nvidia"
make \
    MPI=ompi \
    CC=/usr/bin/gcc-14 \
    CRAY_LIBFABRIC_DIR=/opt/cray/libfabric/2.3.1 \
    CRAY_MPICH_DIR="${OMPI_DIR}" \
    CRAY_PMI_DIR="${OMPI_DIR}" \
    GPU=NVIDIA CUDA_PATH=/opt/nvidia/hpc_sdk/Linux_aarch64/26.3/cuda/13.1

The build creates an executable named xthi in the current working directory. Since the directory is volume mounted from the host, the executable also exists on your host filesystem at $WORKSPACE/xthi.

[!IMPORTANT] Keep the session open and proceed to Create Run-Time Container Image to create a run-time container image for the xthi application.

Create Run-Time Container Image

The next step is to create a run-time image that contains the xthi executable and its runtime dependencies.

Use the SUPS find-required-rpms tool to find the minimum RPM set needed for execution. This helps keep the image small. The tool is pre-installed in the SUPS reference build container image. See SUPS Utility Tools - find-required-rpms for more information about the tool.

printf "%s\n" "xthi" | find-required-rpms --transitive-so

The find-required-rpms tool prints the required RPM package list. Save that list, then exit the build container.

exit

Copy the template files for creating the run-time container image for the containerized application to the workspace.

cp "${SUPS_SOURCE_DIR}/stack/${STACK}/${DISTRO}/run-time/"* "${WORKSPACE}"

The template Apptainer definition file run-time.def will be used to create the run-time container image for xthi using Apptainer.

Add the following line at the end of the %files section to copy in the xthi executable into the container image.

    ./xthi /usr/local/bin/xthi

Then add the following lines at the end of the %post section to install the required RPMs.

For Rocky Linux 9.6 containers

    dnf --assumeyes install \
        <RPMs reported by find-required-rpms>

For SLES 15 SP7 containers

    zypper --non-interactive install \
        <RPMs reported by find-required-rpms>

Build the run-time container image using the modified Apptainer definition file.

apptainer build --fakeroot --build-arg "SUPS_VERSION=${SUPS_VERSION}" \
    --bind "${SUPS_COMMON_DIR}:/var/sups/common:ro" \
    xthi-runtime.sif run-time.def

Run the Containerized Application

[!IMPORTANT] Copy the SIF file to a shared file system that all compute nodes can access (for example, a Lustre file system). The rest of the section assumes you are still in the workspace directory and it can be accessed from all compute nodes in the allocation.

This section covers two ways to launch the containerized xthi application:

• Cray MPICH: uses the SUPS utility tool hpc-container-wrapper with --mpi=cray_shasta. It automatically handles PALS spool directory bind-mounts, Slingshot environment variable forwarding, GPU device and driver-library injection, and GPU isolation. See SUPS Utility Tools - hpc-container-wrapper for more information.

• Open MPI: uses hpc-container-wrapper --mpi=ompi with Slurm --mpi=pmix. The wrapper supplies PMIx connectivity, GPU access and isolation, and Slingshot CXI integration.

Local Sanity Test (Login Node)

Before submitting a multi-node job, verify that the runtime container works correctly on the login node. Login nodes typically have no Slingshot CXI hardware, so exclude the CXI provider to avoid fi_domain() errors at startup.

apptainer exec --env "FI_PROVIDER=^cxi" xthi-runtime.sif /usr/local/bin/xthi

Each MPI rank should print a Hello from rank line. No GPU output is expected since the login node runs a single MPI rank with no Slurm allocation.

Slurm Workload Manager (Slurm)

In this section, we will show how to run the containerized xthi application with Slurm using Apptainer as the container engine. Note that the example is using the Slurm interactive mode, the same commands can be used with Slurm in batch mode as well.

Start an interactive Slurm allocation. Add partition or other Slurm options as needed. This example requests two nodes with two tasks per node.

salloc --nodes=2 --ntasks=4 --ntasks-per-node=2 --distribution=nopack

Cray MPICH

Run the following command to launch the containerized xthi application using the Apptainer SIF file xthi-runtime.sif generated in the previous step.

srun hpc-container-wrapper \
    apptainer run --env OMP_NUM_THREADS=2 \
    xthi-runtime.sif /usr/local/bin/xthi

The containerized xthi application should print output for each MPI rank and OpenMP thread. Example output of an AMD GPU enabled build:

Hello from rank 0, thread 0, on pinoak0011. (core affinity = 0-127)
Hello from rank 0, thread 1, on pinoak0011. (core affinity = 0-127)
Hello from rank 1, thread 0, on pinoak0011. (core affinity = 0-127)
Hello from rank 1, thread 1, on pinoak0011. (core affinity = 0-127)
Hello from rank 2, thread 0, on pinoak0012. (core affinity = 0-127)
Hello from rank 2, thread 1, on pinoak0012. (core affinity = 0-127)
Hello from rank 3, thread 0, on pinoak0012. (core affinity = 0-127)
Hello from rank 3, thread 1, on pinoak0012. (core affinity = 0-127)

*******************
* AMD Device Info *
*******************
rank: 0, device: 0, name: AMD Instinct MI250X, uuid: cdb9d8bb-5212bacd-ea25da50-45ba46c5
rank: 1, device: 0, name: AMD Instinct MI250X, uuid: 20cb1b22-0137440e-5d39399e-f905cbf1
rank: 2, device: 0, name: AMD Instinct MI250X, uuid: 92ca8a54-b8369dc9-e9077a78-2260e9ee
rank: 3, device: 0, name: AMD Instinct MI250X, uuid: e3806bcc-75d5c7fe-0da034e7-4d26850b

Open MPI

With SUPS-built Open MPI, use hpc-container-wrapper --mpi=ompi and launch with Slurm --mpi=pmix. The wrapper provides the PMIx rendezvous mounts and environment, GPU device and driver-library injection, per-rank GPU isolation, and Slingshot CXI integration. When --mpi is omitted, the wrapper auto-detects the MPI implementation from the launcher environment (--mpi=pmix is detected as Open MPI). See SUPS Utility Tools - hpc-container-wrapper for more information.

No image loading is needed: the SIF is read directly from the shared filesystem by every node.

Launch across all ranks of the allocation made above:

srun --ntasks=4 --ntasks-per-node=2 --mpi=pmix \
    hpc-container-wrapper --mpi=ompi \
    apptainer run --env OMP_NUM_THREADS=2 \
        xthi-runtime.sif /usr/local/bin/xthi

[!NOTE] On systems where the Slingshot fabric uses opt-in VNIs (for example, Slurm configured with single_node_vni=user / job_vni=user), add --network=single_node_vni for a single-node job or --network=job_vni for a multi-node job to the srun above. Without a VNI the libfabric CXI provider aborts at startup with fi_domain: Function not implemented; hpc-container-wrapper prints a warning when it detects Slingshot CXI devices but no VNI. If your site auto-allocates VNIs, no flag is needed.

[!NOTE] The salloc shown in this section is CPU-generic and requests no GPUs. A GPU-enabled run must request them in both the allocation and the step — add --gres=gpu:2 to the salloc above as well as to the srun — because a Slurm step can only consume GRES that its parent job already allocated.

The output should be similar to the example output shown in the Cray MPICH section above.

PBS Professional (PBS Pro)

In this section, we will show how to run the containerized xthi application with PBS Pro using Apptainer as the container engine. Note that the example is using the PBS Pro interactive mode, the same commands can be used with PBS Pro in batch mode as well.

Start an interactive PBS Pro allocation. Add partition or other PBS Pro options as needed. This example requests two nodes.

qsub -I -l select=2,place=scatter

Once you are in the interactive session, locate the Cray PALS mpiexec command and make sure it is in your PATH. For example, if Cray PALS 1.9 is installed on your system, you can run the following command:

export PATH="/opt/cray/pals/1.9/bin:${PATH}"

Change into the workspace directory if you are not already in it.

cd "${WORKSPACE}"

Cray MPICH

Run the following command to launch the containerized xthi application using the Apptainer SIF file xthi-runtime.sif generated in the previous step.

mpiexec --no-transfer --np 4 hpc-container-wrapper \
    apptainer exec --env OMP_NUM_THREADS=2 \
    xthi-runtime.sif /usr/local/bin/xthi

The output should be similar to the example output shown in the Cray MPICH section above.

Open MPI

With SUPS-built Open MPI, use hpc-container-wrapper --mpi=ompi. The wrapper supplies the required PMIx, GPU, and Slingshot settings.

mpiexec --no-transfer --np 4 \
    hpc-container-wrapper --mpi=ompi \
    apptainer run --env OMP_NUM_THREADS=2 \
        xthi-runtime.sif /usr/local/bin/xthi

The output should be similar to the example output shown in the Cray MPICH section above.