Installation for user-built application containers

This document describes how to deploy HPE Supercomputing Programming Software (SUPS) for sites where users build their own application containers.

During installation, you will set shell variables that are referenced throughout the installation instructions. These shell variables capture the details of your SUPS deployment, and make it possible to copy and paste most commands without modification.

[!important] If you encounter issues during the build process, refer to the Troubleshooting Guide for guidance.

Choose installation options

SUPS currently supports the Apptainer container engine. Set CONTAINER_ENGINE to the following value.

# To build Apptainer containers.
CONTAINER_ENGINE=apptainer

SUPS builds containers for the following software stacks:

  • AMD software stack

    • Intended for platforms with AMD CPUs and AMD GPUs.

    • Can also be used on platforms with AMD CPUs and no GPUs.

    • Only available on x86_64 CPUs.

  • NVIDIA software stack

    • Intended for platforms with NVIDIA CPUs and NVIDIA GPUs.

    • Can also be used on platforms with NVIDIA CPUs and no GPUs.

    • Only available on aarch64 CPUs.

Choose one of the following values for STACK.

# To build the AMD software stack.
STACK=amd

# To build the NVIDIA software stack.
STACK=nvidia

[!important] You can only build the AMD software stack on x86_64 CPUs, and you can only build the NVIDIA software stack on aarch64 CPUs.

SUPS containers are based on one of the following Linux distributions:

  • Rocky Linux 9.6

  • SUSE Linux Enterprise Server 15 SP7 (SLES 15 SP7)

Choose one of the following values for DISTRO.

# To build Rocky Linux 9.6 containers.
DISTRO=rocky

# To build SLES 15 SP7 containers.
DISTRO=sles

[!important] This determines the Linux distribution within the SUPS containers. The Linux distribution within the container does not need to match the Linux distribution on the host system.

Temporary storage requirements

The SUPS installation process requires a large amount of temporary disk space, up to 250GB depending on the stack being built. The temporary disk space can only be on local storage, and cannot be on a network file system (NFS) or other shared storage like Lustre. Please identify a local storage location on the node where you will run the SUPS installation, ensure that it has sufficient free space, and then follow the instructions below to configure the container engine to use that local storage location as temporary storage during the SUPS installation process. If the node has no local disk at all, see Configure temporary storage on a node with no local disk for a supported alternative.

Set SUPS_TMPDIR to the local storage location you identified. This shell variable is used by the rest of these instructions, and is not read by SUPS itself.

SUPS_TMPDIR=/path/to/local/disk/tmp/dir

Confirm that the location is a local disk with sufficient free space by running the following commands.

findmnt -T "$SUPS_TMPDIR"  # FSTYPE must not be nfs*, gpfs, lustre or panfs
df -h "$SUPS_TMPDIR"       # confirm free space

[!important] If the temporary storage location is a local disk with insufficient free space, you can work around this limitation by reducing the number of parallel container builds during the SUPS installation. Please continue with the installation process, and find the workaround instructions in the Run the build process section.

Configure Apptainer temporary directory

Apptainer uses a temporary directory for building containers. By default, the temporary directory is /tmp. If /tmp doesn’t have sufficient free space, you may configure Apptainer to use a different temporary directory by setting the TMPDIR environment variable.

# Override the default temporary directory used by Apptainer
export TMPDIR="$SUPS_TMPDIR"

Apptainer cache directory

Apptainer keeps a cache of image layers and blobs, separately from the temporary directory above. By default the cache is $HOME/.apptainer/cache, which on most clusters is a quota-limited network filesystem.

The package builds handle this for you. The SUPS build tooling gives each package its own APPTAINER_CACHEDIR beneath its temporary directory, so that parallel builds pulling the same base image do not race each other. That temporary directory is created under TMPDIR, which you set above, and is deleted when the build finishes.

The final reference image is built separately and inherits your environment instead, so it uses APPTAINER_CACHEDIR if you export one and $HOME/.apptainer/cache otherwise. That build bootstraps from a local image rather than a registry, so it downloads no layers, but it still initializes the cache directory. If HOME is unsuitable — over quota, read-only, or on a filesystem you would rather keep clear — export a local location before starting the installation.

# Optional: place the Apptainer cache on local storage
export APPTAINER_CACHEDIR="$SUPS_TMPDIR/apptainer-cache"
mkdir -p "$APPTAINER_CACHEDIR"

The default cache also applies when you pull or build images yourself outside the installation, for example an apptainer pull from a remote registry.

Network filesystem limitations

Container runtimes that remap user and group IDs have a documented limitation on network filesystems: the NFS protocol has no concept of user namespaces, so the UID/GID remapping can fail during operations such as installing packages inside a container. GPFS and Lustre have the same limitation, since neither understands user namespaces either.

For Apptainer this affects --fakeroot and writable sandbox directories, which Apptainer’s own documentation states are not supported on Lustre, GPFS or PanFS and which return permission errors on NFS. It also applies to TMPDIR/APPTAINER_TMPDIR while building. Reading a finished SIF from a shared filesystem is unaffected — only unpacked sandboxes and build temporary directories need to be on local storage such as SUPS_TMPDIR.

Configure temporary storage on a node with no local disk

If no local storage is available, you can create one with a loopback-mounted image file. This is an exception to the local storage requirement above: a loop device presents the image file as a block device carrying a real local filesystem, so the operations performed while building containers work normally, even though the backing file itself is on shared storage.

The following example creates a 250GB image on a Lustre filesystem, formats it as XFS, and mounts it at $SUPS_TMPDIR, the temporary directory configured above.

# Create a 250GB sparse image on a Lustre filesystem
truncate -s 250g /lustre/sups/sups-storage.img

# Format the image as XFS
mkfs.xfs -m reflink=1 /lustre/sups/sups-storage.img

# Mount the image at the temporary storage location configured above
mkdir -p "$SUPS_TMPDIR"
sudo mount -o loop /lustre/sups/sups-storage.img "$SUPS_TMPDIR"

# Give the mount point to the user who will run the SUPS installation
sudo chown "$(id -u):$(id -g)" "$SUPS_TMPDIR"

The mount command requires root privileges, and the mount does not survive a reboot. If the node is rebooted before the SUPS installation completes, run the mount and chown commands again.

Choose installation location

There are five directories involved in a SUPS deployment: the common directory, the download directory, the image directory, the log directory, and the source directory. These are described in Directory structure.

To enable users to build their own application containers, the common directory, image directory, and source directory must be accessible to users. Therefore, it is recommended (but not required) to put these directories on shared storage, under the same top-level directory.

Choose values for the following variables.

# To put the common, image, and source directories under the same
# top-level directory.
COMMON_TOP_DIR=/path/to/top/dir
IMAGE_TOP_DIR=/path/to/top/dir
SOURCE_TOP_DIR=/path/to/top/dir

Users do not need access to the download directory or the log directory. Therefore, it is recommended (but not required) to put them in a private location on shared storage (for example, under your home directory). You may also choose to put the download and log directories under the same top-level directory as the other three directories above.

Choose values for DOWNLOAD_TOP_DIR and LOG_TOP_DIR.

# To put the download and log directories under your home directory.
DOWNLOAD_TOP_DIR="$HOME/sups"
LOG_TOP_DIR="$HOME/sups"

Install run-time prerequisites

Install SUPS run-time prerequisites in all login and compute node images. Adapt the following commands to suit your cluster manager.

GPU driver requirements

This section applies only to nodes with GPU hardware. SUPS does not require a GPU, and both software stacks are supported on CPU-only nodes, as described in Choose installation options.

On such nodes, a working GPU driver must already be installed and correctly configured on the host. This is a system administration task to complete before starting running containerized GPU applications:

  • For the AMD software stack, install the AMD GPU (ROCm amdgpu) driver.

  • For the NVIDIA software stack, install the NVIDIA GPU (CUDA) driver.

The driver version installed on the host must be compatible with both the GPU hardware available on the host, as well as the host operating system. Refer to your distribution’s documentation for driver installation steps.

For more information on host software installation and configuration, see the AMD ROCm installation guide for the AMD software stack, or the NVIDIA Driver Installation Guide for Linux for the NVIDIA software stack.

Red Hat Enterprise Linux and Rocky Linux

For Red Hat Enterprise Linux (RHEL) and Rocky Linux nodes, install the following packages in all login and compute node images.

dnf install python3

Install Apptainer in all login and compute node images. Apptainer is available from the Extra Packages for Enterprise Linux (EPEL) repository.

dnf install epel-release
dnf install apptainer

Install the unprivileged apptainer package, not apptainer-suid. SUPS user documentation assumes that apptainer will be run rootless with --fakeroot, using a non-setuid Apptainer installation.

If your nodes have AMD GPU hardware, follow AMD ROCm’s instructions to add the ROCm RPM repository to your system, then install AMD System Management libraries from the ROCm RPM repository. Install this in all login and compute node images with AMD GPU hardware.

# For ROCm 7.2 and earlier
dnf install rocm-smi-lib

# For ROCm 7.13 and later
dnf install amdrocm-amdsmi

SUSE Linux Enterprise Server

For SUSE Linux Enterprise Server (SLES) nodes, install the following packages in all login and compute node images.

zypper install python3

Install Apptainer in all login and compute node images. In SLES 15 SP7, Apptainer is available from the Module-HPC-update repository.

zypper install apptainer

Install the unprivileged apptainer package, not apptainer-suid. SUPS user documentation assumes that apptainer will be run rootless with --fakeroot, using a non-setuid Apptainer installation.

If your nodes have AMD GPU hardware, follow AMD ROCm’s instructions to add the ROCm RPM repository to your system, then install AMD System Management libraries from the ROCm RPM repository. Install this in all login and compute node images with AMD GPU hardware.

# For ROCm 7.2 and earlier
zypper install rocm-smi-lib

# For ROCm 7.13 and later
zypper install amdrocm-amdsmi

Unpack the source directory

Complete these steps on a login node or compute node.

First, set the following shell variables that are used throughout the rest of these instructions.

If you are deploying SUPS from a tarball, set TARBALL.

# Example SUPS tarball.
TARBALL="/path/to/sups-0.0.0-alpha3-30.b8cc91f.tar.gz"

If you are deploying SUPS from an RPM, set RPM.

# Example SUPS RPM.
RPM="/path/to/sups-0.0.0~alpha3^30.b8cc91f-1.noarch.rpm"

Set VERSION to the SUPS version.

# Example SUPS version.
VERSION="0.0.0~alpha3^30.b8cc91f"

Overwrite VERSION using the following command. The new value is the same as before, but with special characters replaced by hyphens.

VERSION="$(echo "$VERSION" | tr -c "[:alnum:].\n" -)"

Unpack SUPS source files under $SOURCE_TOP_DIR.

# Use these commands if you are unpacking a SUPS tarball.
SOURCE_DIR="$SOURCE_TOP_DIR/source/$VERSION"
mkdir -p "$SOURCE_DIR"
tar -C "$SOURCE_DIR" --strip-components 1 -xaf "$TARBALL"

# Use these commands if you are unpacking a SUPS RPM.
mkdir -p "$SOURCE_TOP_DIR/source" "$SOURCE_TOP_DIR/tmp"
rpm2cpio "$RPM" | cpio -iVdD "$SOURCE_TOP_DIR/tmp"
mv -t "$SOURCE_TOP_DIR/source" "$SOURCE_TOP_DIR/tmp/opt/cray/sups/source/$VERSION"
rm -r "$SOURCE_TOP_DIR/tmp"

Download required files

Follow the instructions in Download required files to download required files. Complete these steps on a login node or compute node.

Install build prerequisites

Install SUPS build prerequisites on a login node or compute node. The build prerequisites are only required during the SUPS build process, so they do not need to be permanently installed in the node image.

Red Hat Enterprise Linux and Rocky Linux

For Red Hat Enterprise Linux (RHEL) and Rocky Linux nodes, install the following packages.

dnf install \
    alternatives \
    coreutils \
    createrepo_c \
    curl \
    python3 \
    tar \
    yum-utils

Install Apptainer, available from the Extra Packages for Enterprise Linux (EPEL) repository.

dnf install epel-release
dnf install apptainer

SUSE Linux Enterprise Server

For SUSE Linux Enterprise Server (SLES) nodes, install the following packages.

zypper install \
    coreutils \
    createrepo_c \
    curl \
    python3 \
    tar \
    update-alternatives

Install Apptainer, available in SLES 15 SP7 from the Module-HPC-update repository.

zypper install apptainer

Run the build process

Run the SUPS build process on the same node where you installed the build prerequisites.

[!important] By default, the build process uses all but one of the CPUs on the build node for parallel compilation jobs. If the build node is being used for other activity at the same time, this can cause the system to become unresponsive or crash. If the build node is shared or otherwise in use, pass --max-nproc N to tools/install, where N is the maximum number of CPUs to use, to limit the number of parallel compilation jobs across all in-progress build containers.

[!important] If the temporary storage location has insufficient free space, you can work around this limitation by reducing the number of parallel container builds during the SUPS installation. Pass --max-parallel M to tools/install, where M is the maximum number of parallel container builds. For example, if the temporary storage location has about 200GB of free space, you can set M to 8 and reduce it again if you still encounter disk space issues.

# Run the SUPS build process.
"$SOURCE_TOP_DIR/source/$VERSION/tools/install" \
    --common-top-dir "$COMMON_TOP_DIR" \
    --container-engine "$CONTAINER_ENGINE" \
    --distro "$DISTRO" \
    --download-top-dir "$DOWNLOAD_TOP_DIR" \
    --image-top-dir "$IMAGE_TOP_DIR" \
    --log-top-dir "$LOG_TOP_DIR" \
    --stack "$STACK"

After the build process completes, you may delete the download directory. You may also choose to keep the download directory for future reference.

# Delete the download directory (optional).
rm -r "$DOWNLOAD_TOP_DIR/download"

Post-installation steps

If you have to move the SUPS directories to a different location after the build process, do it before you start the post-installation steps and set the corresponding shell variables to the new locations.

# If the common directory has been moved to a new location
COMMON_TOP_DIR=/new/path/to/common/top/dir

# If the image directory has been moved to a new location
IMAGE_TOP_DIR=/new/path/to/image/top/dir

# If the source directory has been moved to a new location
SOURCE_TOP_DIR=/new/path/to/source/top/dir

After the build process completes, you will need to make the SUPS host utility tools in $SOURCE_TOP_DIR/source/$VERSION/bin available in the user’s PATH environment variable.

SUPS offers a post-install script that helps you do this by installing the SUPS binaries in /usr/bin and registering them with the system’s alternatives mechanism. Using the SUPS post-install script is optional, and you may choose to make the SUPS binaries available in the user’s PATH environment variable by other means.

# Run the SUPS post-install script.
"$SOURCE_TOP_DIR/source/$VERSION/tools/post-install"

Run the sups-config utility tool to verify that the SUPS configuration file is valid and that all directories referenced by the configuration exist.

# Run the sups-config utility tool.
"$SOURCE_TOP_DIR/source/$VERSION/bin/sups-config"

# Or if the SUPS binaries are already in PATH, simply run
sups-config

If the tool reports any invalid configuration, you can use the tool to update the configuration file. You don’t have to update all the following variables, only the ones that are invalid.

# Update the common directory location
sups-config --set SUPS_COMMON_DIR="$COMMON_TOP_DIR/common"

# Update the image directory location
sups-config --set SUPS_IMAGE_DIR="$IMAGE_TOP_DIR/image/$VERSION"

# Update the source directory location
sups-config --set SUPS_SOURCE_DIR="$SOURCE_TOP_DIR/source/$VERSION"

Run the sups-config utility tool again to verify that everything is valid.

Configure SUPS

Finish the following optional configuration steps to properly set up SUPS and other dependent components.

Configure rootless Apptainer

You may need to configure the “rootless” fakeroot feature for Apptainer.

Please refer to the Apptainer documentation “Rootless” Fakeroot feature for details.

Pyxis administration

Pyxis is the SPANK plugin for Slurm that allows unprivileged users to run containerized tasks with srun --container-image=<image>. Configuring the plugin, bind-mounting the SUPS environment wrappers, setting up task prologs, and configuring GPU isolation are covered in a dedicated guide.

See the Pyxis Administrator Guide for details.

Next steps

SUPS is now installed and ready for users to build their own application containers. Direct your users to the following guides.