Guidance on Building or Running CPE on a Container
CPE Container Options
Alongside our official release on our support site https://myenterpriselicense.hpe.com/cwp-ui/software, we also supply a set of buildable Dockerfiles mimicking a complete installation of CPE with all required dependencies.
Containers allow you to set up and configure environments for building, running and profiling HPE HPC applications and distribute them using images:
Install an image containing an environment pre-configured with all the tools you need, then develop within that environment.
Save an environment and use the image to move that environment to another machine without additional setup.
Customize and prepare containers with different sets of languages and runtimes, analysis tools, or other tools, as needed.
These are experimental examples, please provide any feedback to our Slack Channel.
No Official Support
Please note that using this method of downloading, installing, configuring, or building software is not officially supported. Please join our Slack Workspace and start a discussion in our #hpe-cray-programming-environment channel.
Building a CPE container
The CPE container release distributes example Dockerfiles in the directory build/dockerfiles that can be used to install a full CPE release for a given OS and architecture.
The example Dockerfiles will by default expect to install content from the CPE repo on HPESC, leaving blank fields to fill out for an HPESC access token:
# Replace with your base OS source image, SLES or Red Hat
FROM <sles_image> as base
# Your HPESC email. Replace @ with %40
# For example, user@hpe.com would become `ARG HPESC_USER=user%40hpe.com`
ARG HPESC_USER=<user>
ARG HPESC_TOKEN=<token>
# No need to modify these lines, they are filled in using the above arguments
RUN rpm --import https://${HPESC_USER}:${HPESC_TOKEN}@update1.linux.hpe.com/repo/cpe/HPE-RPM-PROD-KEY-FIPS.public
RUN zypper addrepo -f https://${HPESC_USER}:${HPESC_TOKEN}@update1.linux.hpe.com/repo/cpe/%VERSION%/base/sle/15.6/ cpe-%VERSION%
If applicable, edit Dockerfiles as needed to source packages from your site’s local CPE mirror:
RUN rpm --import https://your.site.mirror/cpe/HPE-RPM-PROD-KEY-FIPS.public
RUN zypper addrepo -f https://your.site.mirror/cpe/%VERSION%/base/sle/15.6/ cpe-%VERSION%
USS package sources
HPE USS packages provide certain runtime dependencies for CPE components. As of the current release, USS packages are not available via the HPESC repo, and can instead be installed from the USS repo ISO.
The provided Dockerfiles expect USS content provided at the containerized directory /ussrepo/:
RUN zypper addrepo -f --no-gpgcheck dir:///ussrepo/uss-wlm && zypper refresh uss-wlm
If you have access to the USS repo ISO, mount it on the host system, then build the container image with:
cd HPE_CPE_Container_%VERSION%/build/
mkdir -p mnt
mount -o loop uss-wlm-support-<version>-<os>-<arch>.iso mnt
podman build -t ccpe -f dockerfiles/<os>_<arch>.Dockerfile \
-v $PWD/mnt:/ussrepo:ro --network host .
If you have other sources for the USS packages, you can remove the uss-wlm repo configuration from the Dockerfile, and replace it with a manual install of the USS RPMs:
ADD /path/to/uss-rpms/*.rpm /rpms/
RUN rpm -idv --nodeps /rpms/*.rpm
If manually installing RPMs, a bindmount of the USS repo directory is not needed during build:
podman build -t ccpe -f dockerfiles/sles15_sp6_x86_64.Dockerfile \
--network host .
Create a container using ccpe-config
To inherit the host’s Slurm environment, a script ccpe-config is provided to generate
the necessary Podman arguments. Running ccpe-config will generate the arguments for
podman run to inherit Slurm:
# Start CCPE container
podman run -d -it --name ccpe `./ccpe-config` cpe-sles15-sp6-x86-64:latest
After starting the container, you can attach to it for an interactive session:
podman attach ccpe
You can also run a command directly in the container. For this case, you must
wrap your command in the container’s entrypoint /ccpe-entry. This ensures that the
proper environment is set up and propagated to your application:
podman exec -it ccpe /ccpe-entry a.out arg1 arg2
For Singularity, ccpe-config -s will generate Singularity arguments.
Root access inside the container is not supported with Singularity.
singularity run `./ccpe-config -s` ccpe.sif
You can also generate a podman-compose file:
./ccpe-config -c
podman-compose up --detach
Attach to the running bash session inside the container
The entrypoint to the Podman container sets up the proper permissions to run as your local user,
but with the ability to run commands with sudo to manage the container:
podman attach ccpe
user@ccpe-container:~> module list
Currently Loaded Modulefiles:
1) cce/20.0.0 5) PrgEnv-cray/8.6.0 9) libfabric/2.2.0rc1
2) craype/2.7.35 6) craype-x86-spr 10) xpmem/1.0.1-1.4_gf67ddfb318d4
3) cray-dsmml/0.3.1 7) craype-network-ofi
4) cray-libsci/25.09.0 8) cray-mpich/9.0.1
user@ccpe-container:~> sudo whoami
root
Stop the container and delete it:
podman stop ccpe
podman rm ccpe
You can also generate a podman-compose file:
./ccpe-config -c
podman-compose up --detach
Singularity
For Singularity, ccpe-config -s will generate Singularity arguments.
Root access inside the container is not supported with Singularity.
# Export container as tarfile
podman save --format docker-archive cpe-sles15-sp6-x86-64:latest -o ccpe.tar
# Create Singularity SIF from tarfile
singularity build ccpe.sif docker-archive:ccpe.tar
# Start an interactive session
singularity run `./ccpe-config -s` ccpe.sif
# Run a command directly
# Wrap command in the entrypoint script `ccpe-entry`
singularity exec `./ccpe-config -s` ccpe.sif /ccpe-entry a.out arg1 arg2
# Run a containerized MPI job under Slurm
# Wrap command in the entrypoint script `ccpe-entry`
srun -N2 -n4 singularity exec `./ccpe-config -s` ccpe.sif /ccpe-entry a.out arg1 arg2
Launch containerized job onto compute nodes
if you would like to run a containerized application, with one container instance per rank, select this option. Ensure that the container image is accessible from all nodes:
# Run a containerized MPI job in Singularity container
srun -N2 -n4 singularity exec `./ccpe-config -s` ccpe.sif /ccpe-entry a.out arg1 arg2
Launch containerized job using SBATCH
ccpe-config can generate a system-specific SBATCH file for launching Slurm batch jobs:
./ccpe-config --sbatch > application.sbatch
This batch file will automatically set system mounts and environment variables for launching a Singularity container hosting your application on multiple compute nodes.
At the top of the generated file, you will see:
#!/bin/bash
#SBATCH -n <nranks>
# Other SBATCH arguments go here
# Edit this setting for the image path:
SIF_PATH=<Singularity image path, accessible from compute nodes>
# Job launch should be configured at the bottom of this generated file
First, change and add #SBATCH parameters as needed for your system and job.
Next, set SIF_PATH to the path to your built CCPE Singularity image file.
Finally, at the bottom of the generated batch script is the actual Singularity run line:
srun singularity exec $singularity_args $SIF_PATH /ccpe-entry /bin/bash -C << 'SIF_EOF'
# Job run line. Add launch commands here, ensuring command list ends with SIF_EOF
echo Hello from inside container!
SIF_EOF
This is a multi-line command list that will be launched inside the container.
For example, replace the placeholder echo command with your target MPI application.
You can also add logging here, and use Bash variables normally within the scope of the run line:
srun singularity exec $singularity_args $SIF_PATH /ccpe-entry /bin/bash -C << 'SIF_EOF'
# Use containerized PE
cc -g -O2 source.c -o a.out
a_out_settings="arg1 arg2"
./a.out $a_out_settings arg3 arg4
SIF_EOF
Launch uncontainerized jobs from single container instance
If you would like to run a single copy of the PE container for an interactive session, and run your application uncontainerized on compute nodes, select this option. The application can be built with container dependencies. Libraries present in container will be automatically shipped to nodes. It is not required for the container image to be accessible from the nodes:
# Launch PE container with Podman
podman run -d -it --name ccpe `./ccpe-config` ccpe-devel-sles15sp5-x86_64:latest
# Inside container, run application with automatic dependency shipping
user@ccpe-container:~> cc containerized_application.c -o a.out
user@ccpe-container:~> srun -N2 -n4 ./a.out
# Launch PE container with Singularity
singularity run `./ccpe-config -s` ccpe.sif
# Inside container, run application with automatic dependency shipping
Apptainer> cc containerized_application.c -o a.out
Apptainer> srun -N2 -n4 ./a.out
ccpe-config will inherit the host’s Slurm configuration, and set additional options to allow running
Slurm jobs built inside the container. Slurm will automatically ship out dynamic library dependencies
for the target application. Jobs will run directly on the compute nodes, and not inside the container.
Additional dependencies that are e.g. not present in the shared home directory can be added to the job by setting the option:
SLURM_BCAST_EXTRA=/path/to/other/dependency.so,/second/dependency.so
Extra files are shipped to the directory defined using Slurm job-specific environment variables:
${SLURM_BCAST}/slurm_bcast_${SLURM_JOBID}.${SLURM_STEPID}_${SLURMD_NODENAME}_libs/
For example:
import sys
import os
bcast = os.getenv("SLURM_BCAST")
jobid = os.getenv("SLURM_JOBID")
stepid = os.getenv("SLURM_STEPID")
nodename = os.getenv("SLURMD_NODENAME")
libs_dir = f'{bcast}/slurm_bcast_{jobid}.{stepid}_{nodename}_libs'
with open(f'{libs_dir}/file.txt') as file:
for line in file:
print(line)
user@ccpe-container:~> SLURM_BCAST_EXTRA=/path/in/container/file.txt srun python3 ./ls.py
Contents of file.txt
Runtime customization
By default, ccpe-config will create new containers inheriting system GPU SDKs from /opt/nvidia
and /opt/rocm-*. GPU SDKs are inherited (instead of installed in the container) to ensure that
they are compatible with host GPU drivers.
Additional GPU modulefiles are be inherited from /opt/modulefiles/nvidia and /opt/modulefiles/amd.
The locations of these directories from the host sytem can be changed by modifying mount_files in
the ccpe-config script.
Additionally, MODULEPATH can be updated by changing the MODULEPATH entry in shell_env_settings
to point to the correct inherited module paths.