Spack User documentation

Overview

The following assumes the user has a basic level of Spack experience. This documentation is intended to provide a schematic to setup compilers and software packages available in the HPE Cray Programming Environment (CPE). For issue related to Spack, it is encouraged to consult the official Spack documentation.

CPE Compiler specification

It is recommended that any compiler be invoked without the CPE compiler drivers . Unlike the CPE compiler driver approach, the “driver-less” compiler invocation does not append tooling to use MPI or LIBSCI. This avoids unintended behavior and gives Spack complete control over the build trajectory. Unlike the CPE compiler drivers which unify the compilers under the same executables (e.g. cc/CC/ftn), for the “driver-less” approach this is no longer the case. For example, the following table demonstrates the differences for compiler executables for several compilers:

Compiler

C

C++

Fortran

CCE

craycc

crayCC

crayftn

GCC

gcc

g++

gfortran

ROCmCC

amdclang

amdclang++

amdflang

Intel

icc

icpc

ifort

nvhpc

nvc

nvc++

nvfortran

oneapi

icx

icpx

ifx

aocc

clang

clang++

flang

As an example, the CCE 17.0.1 compiler can be configured by the following yaml code:

compilers:
- compiler:
    spec: cce@=17.0.1
    paths:
      cc: /opt/cray/pe/cce/17.0.1/bin/craycc
      cxx: /opt/cray/pe/cce/17.0.1/bin/craycxx
      f77: /opt/cray/pe/cce/17.0.1/bin/crayftn
      fc: /opt/cray/pe/cce/17.0.1/bin/crayftn
    flags: {}
    operating_system: sles15
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []

The compiler configuration for the CCE 17.0.1 specifies the full paths to craycc/crayCC/crayftn. Additional fields can be populated to control compiler flags, environment variables, etc… Typically the most difficult take to populate a compiler configuration is find the paths to the various compilers. On CrayEX systems, the location of the driver-less compilers are found under /opt/cray/pe/cce/<CCE version>/bin. If you are unsure where to look for craycc/crayCC/crayftn you can try to use “which craycc” to locate the bin directory for the compilers. Note that it is important that the “operating_system:” files matches the os reported by “spack arch”,

A set of compilers can be specified by appending a separate “- compiler:” field for each compiler to the “compilers:” field. The following demonstrates how three separate compilers are specified (e.g. CCE 17.0.1, ROCmCC 6.0.0, and GCC 13.3.0):

compilers:
- compiler:
    spec: cce@=17.0.1
    paths:
      cc: /opt/cray/pe/cce/17.0.1/bin/craycc
      cxx: /opt/cray/pe/cce/17.0.1/bin/craycxx
      f77: /opt/cray/pe/cce/17.0.1/bin/crayftn
      fc: /opt/cray/pe/cce/17.0.1/bin/crayftn
    flags: {}
    operating_system: sles15
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: rocmcc@=6.0.0
    paths:
      cc: /opt/rocm-6.0.0/bin/amdclang
      cxx: /opt/rocm-6.0.0/bin/amdclang++
      f77: null
      fc: /opt/rocm-6.0.0/bin/amdflang
    flags: {}
    operating_system: sles15
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@=13.3.0
    paths:
      cc: /opt/cray/pe/gcc-native/13/bin/gcc
      cxx: /opt/cray/pe/gcc-native/13/bin/g++
      f77: null
      fc: /opt/cray/pe/gcc-native/13/bin/gfortran
    flags: {}
    operating_system: sles15
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []

CPE software configuration

Source software for CPE packages (e.g. cray-mpich, cray-libsci, cray-fftw) is generally unavailable. Instead of building, users provide details that specify the package installation location. These packages are considered external in the Spack sense. It’s important to understand that for certain CPE packages there are multiple variants, each targeting a different compiler or hardware architecture. For example, on an arbitrary system with CPE, the cray-mpich package shows 6 different variants (amd, aocc, cray, gnu, intel, nvidia), each to be used with its corresponding compiler.:

> ls -l /opt/cray/pe/mpich/8.1.29/ofi | grep -v ^l
total 0
drwxr-xr-x 1 root root 26 May 23 03:07 amd
drwxr-xr-x 1 root root 26 May 23 03:14 aocc
drwxr-xr-x 1 root root 27 May 23 03:18 cray
drwxr-xr-x 1 root root 27 May 23 03:18 gnu
drwxr-xr-x 1 root root 29 May 23 03:19 intel
drwxr-xr-x 1 root root 27 May 21 03:06 man
drwxr-xr-x 1 root root 27 May 23 03:08 nvidia

Conversely, some CPE packages may show different variants for different hardware architectures, e.g. cray-fftw can have variants for skylake (intel), genoa (amd), etc…

> ls -l /opt/cray/pe/fftw/3.3.10.7/ | grep -v ^l
total 5
drwxr-xr-x 1 root root   64 Jun  6 03:09 broadwell
drwxr-xr-x 1 root root   52 Jun  6 03:09 haswell
-rw-r--r-- 1 root root 1680 Feb  7 09:22 release_info
-rwxr-xr-- 1 root root 1665 Jun  6 03:09 set_default_fftw_3.3.10.7
-rw-r--r-- 1 root root  115 Jun  6 03:09 set_pkgconfig_default_fftw
drwxr-xr-x 1 root root   26 Jun  6 03:09 share
drwxr-xr-x 1 root root   52 Jun  6 03:09 x86_genoa
drwxr-xr-x 1 root root   64 Jun  6 03:09 x86_milan
drwxr-xr-x 1 root root   52 Jun  6 03:09 x86_rome
drwxr-xr-x 1 root root   52 Jun  6 03:09 x86_skylake

To setup CPE packages, the user must ascertain which compiler(s)/hardware-architecture will be used and then find the corresponding install prefixes for the desired CPE package(s). The following yaml source shows configurations for cray-mpich, cray-libsci, and cray-fftw. The cray-fftw package contains a single prefix, which depends on the hardware architecture. Conversely, cray-mpich and cray-libsci contain three unique prefixes, each of which corresponds to a different compiler. This requires three variants be declared in the package configurations. This is accomplished by appending a %<compiler id> to the “- spec:” heading. :

packages:
  cray-mpich:
    buildable: false
    variants: +wrappers
    externals:
    - spec: cray-mpich@8.1.29 %cce
      prefix: /opt/cray/pe/mpich/8.1.29/ofi/cray/17.0
    - spec: cray-mpich@8.1.29 %gcc
      prefix: /opt/cray/pe/mpich/8.1.29/ofi/gnu/12.3
    - spec: cray-mpich@8.1.29 %rocmcc
      prefix: /opt/cray/pe/mpich/8.1.29/ofi/amd/5.0
  cray-libsci:
    buildable: false
    externals:
    - spec: cray-libsci@23.12.5 %cce
      prefix: /opt/cray/pe/libsci/24.03.0/CRAY/17.0/x86_64
    - spec: cray-libsci@23.12.5 %gcc
      prefix: /opt/cray/pe/libsci/24.03.0/gnu/12.3/x86_64
    - spec: cray-libsci@23.12.5 %rocmcc
      prefix: /opt/cray/pe/libsci/24.03.0/amd/5.0/x86_64
  cray-fftw:
    buildable: false
    externals:
    - spec: cray-fftw@3.3.10.7
      prefix: /opt/cray/pe/fftw/3.3.10.7/x86_genoa

Example

Now that you’ve seen how to setup compilers and CPE software packages, let’s go through an illustrative example to demonstrates the interoperability of CPE with Spack. This example use two different compilers to build Trilinos inside of a “Spack Environment”

  1. Setup a spack environment:

    #clone spack
git clone https://github.com/spack/spack.git spack

#activate spack
. spack/share/spack/setup-env.sh

#create a spack environment
spack env create myenv

#activate your environemnt
spack env activate myenv

#navigate to your environment
spack cd -e

  2. Overwrite the spack.yaml  file with the following:

    > cat compilers.yaml
(1) lroskop@pinoak-login1:/lus/cflus02/lroskop/spackTest/var/spack/environments/myenv cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs:
  - scorep%gcc
  - scorep%cce
  view: true
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: cce@=17.0.1
      paths:
        cc: /opt/cray/pe/cce/17.0.1/bin/craycc
        cxx: /opt/cray/pe/cce/17.0.1/bin/craycxx
        f77: /opt/cray/pe/cce/17.0.1/bin/crayftn
        fc: /opt/cray/pe/cce/17.0.1/bin/crayftn
      flags: {}
      operating_system: sles15
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  - compiler:
      spec: rocmcc@=6.0.0
      paths:
        cc: /opt/rocm-6.0.0/bin/amdclang
        cxx: /opt/rocm-6.0.0/bin/amdclang++
        f77: null
        fc: /opt/rocm-6.0.0/bin/amdflang
      flags: {}
      operating_system: sles15
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  - compiler:
      spec: gcc@=13.3.0
      paths:
        cc: /opt/cray/pe/gcc-native/13/bin/gcc
        cxx: /opt/cray/pe/gcc-native/13/bin/g++
        f77: null
        fc: /opt/cray/pe/gcc-native/13/bin/gfortran
      flags: {}
      operating_system: sles15
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  packages:
    cray-mpich:
      buildable: false
      variants: +wrappers
      externals:
      - spec: cray-mpich@8.1.29 %cce
        prefix: /opt/cray/pe/mpich/8.1.29/ofi/cray/17.0
      - spec: cray-mpich@8.1.29 %gcc
        prefix: /opt/cray/pe/mpich/8.1.29/ofi/gnu/12.3

  3. Concretize your environment:

    > spack concretize -f
==> Starting concretization pool with 2 processes
==> Environment concretized in 58.92 seconds
==> Concretized trilinos%gcc
 -   hp6x7kb  trilinos@15.1.1%gcc@13.3.0~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~pamgen~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~test~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_system=cmake build_type=Release cxxstd=17 generator=make gotype=long_long arch=linux-sles15-zen2
 -   x66lc2e      ^cmake@3.29.2%gcc@13.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-sles15-zen2
 -   cwqvvyj          ^curl@8.7.1%gcc@13.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen2
 -   3fizc2b              ^nghttp2@1.61.0%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
 -   wskqo2e                  ^diffutils@3.10%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
 -   eubmydq              ^openssl@3.3.0%gcc@13.3.0~docs+shared build_system=generic certs=mozilla arch=linux-sles15-zen2
 -   qkggq7i                  ^ca-certificates-mozilla@2023-05-30%gcc@13.3.0 build_system=generic arch=linux-sles15-zen2
 -   6kmfkj6          ^ncurses@6.5%gcc@13.3.0~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-sles15-zen2
 -   puzy66t          ^zlib-ng@2.1.6%gcc@13.3.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-sles15-zen2
[e]  4vrb46p      ^cray-mpich@8.1.29%gcc@13.3.0+wrappers build_system=generic arch=linux-sles15-zen2
[+]  tcxrp33      ^gcc-runtime@13.3.0%gcc@13.3.0 build_system=generic arch=linux-sles15-zen2
[e]  raqrfdc      ^glibc@2.31%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
[+]  rzq4pyy      ^gmake@4.4.1%gcc@13.3.0~guile build_system=generic arch=linux-sles15-zen2
 -   qsawojj      ^hwloc@2.9.1%gcc@13.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-sles15-zen2
 -   nlgmgsb          ^libpciaccess@0.17%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
 -   bh3uxgu              ^libtool@2.4.7%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
 -   2ngywdi                  ^findutils@4.9.0%gcc@13.3.0 build_system=autotools patches=440b954 arch=linux-sles15-zen2
 -   p7n6ddt                  ^m4@1.4.19%gcc@13.3.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-sles15-zen2
 -   eutf24o                      ^libsigsegv@2.14%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
 -   gv7nrsi              ^util-macros@1.19.3%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
 -   wdy3trk          ^libxml2@2.10.3%gcc@13.3.0+pic~python+shared build_system=autotools arch=linux-sles15-zen2
 -   6wp25xm              ^xz@5.4.6%gcc@13.3.0~pic build_system=autotools libs=shared,static arch=linux-sles15-zen2
[+]  kr22aje          ^pkgconf@2.2.0%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
[e]  zoxojdx              ^glibc@2.31%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
[+]  ja7yy26              ^gmake@4.4.1%cce@17.0.1~guile build_system=generic arch=linux-sles15-zen2
 -   irkimrp      ^kokkos@4.2.01%gcc@13.3.0~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-sles15-zen2
 -   dzsgkmk      ^openblas@0.3.26%gcc@13.3.0~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-sles15-zen2
 -   zd6pbmf          ^perl@5.38.2%gcc@13.3.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-sles15-zen2
 -   y46ffkn              ^berkeley-db@18.1.40%gcc@13.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-sles15-zen2
 -   xhkgp3h              ^bzip2@1.0.8%gcc@13.3.0~debug~pic+shared build_system=generic arch=linux-sles15-zen2
 -   q5csc26              ^gdbm@1.23%gcc@13.3.0 build_system=autotools arch=linux-sles15-zen2
 -   ophosub                  ^readline@8.2%gcc@13.3.0 build_system=autotools patches=bbf97f1 arch=linux-sles15-zen2

==> Concretized trilinos%cce
 -   cow2rdx  trilinos@15.1.1%cce@17.0.1~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~pamgen~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~test~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_system=cmake build_type=Release cxxstd=17 generator=make gotype=long_long arch=linux-sles15-zen2
 -   hxymsfw      ^cmake@3.29.2%cce@17.0.1~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-sles15-zen2
 -   rayborz          ^curl@8.7.1%cce@17.0.1~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen2
 -   4iomo4y              ^nghttp2@1.61.0%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   kdefr36                  ^diffutils@3.10%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   zbf6s6a              ^openssl@3.3.0%cce@17.0.1~docs+shared build_system=generic certs=mozilla arch=linux-sles15-zen2
 -   wjit3om                  ^ca-certificates-mozilla@2023-05-30%cce@17.0.1 build_system=generic arch=linux-sles15-zen2
 -   cryvdlf          ^ncurses@6.5%cce@17.0.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-sles15-zen2
 -   xytcb4q          ^zlib-ng@2.1.6%cce@17.0.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-sles15-zen2
[e]  3qowoqv      ^cray-mpich@8.1.29%cce@17.0.1+wrappers build_system=generic arch=linux-sles15-zen2
[e]  zoxojdx      ^glibc@2.31%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
[+]  ja7yy26      ^gmake@4.4.1%cce@17.0.1~guile build_system=generic arch=linux-sles15-zen2
 -   l7lwsuz      ^hwloc@2.9.1%cce@17.0.1~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-sles15-zen2
 -   4zb2bak          ^libpciaccess@0.17%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   qmxu5yl              ^libtool@2.4.7%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   qn4znzr                  ^findutils@4.9.0%cce@17.0.1 build_system=autotools patches=440b954 arch=linux-sles15-zen2
 -   i3kicpn                  ^m4@1.4.19%cce@17.0.1+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-sles15-zen2
 -   p7yzsy4                      ^libsigsegv@2.14%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   37jawrj              ^util-macros@1.19.3%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   chhatkg          ^libxml2@2.10.3%cce@17.0.1+pic~python+shared build_system=autotools arch=linux-sles15-zen2
 -   vrl3eu3              ^xz@5.4.6%cce@17.0.1~pic build_system=autotools libs=shared,static arch=linux-sles15-zen2
[+]  kr22aje          ^pkgconf@2.2.0%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   d4uxglw      ^kokkos@4.2.01%cce@17.0.1~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-sles15-zen2
 -   kcdmju3      ^openblas@0.3.26%cce@17.0.1~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-sles15-zen2
 -   q7rfe3p          ^perl@5.38.2%cce@17.0.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-sles15-zen2
 -   g2nghwz              ^berkeley-db@18.1.40%cce@17.0.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-sles15-zen2
 -   x2v545u              ^bzip2@1.0.8%cce@17.0.1~debug~pic+shared build_system=generic arch=linux-sles15-zen2
 -   cpmcpqb              ^gdbm@1.23%cce@17.0.1 build_system=autotools arch=linux-sles15-zen2
 -   6uiwsmw                  ^readline@8.2%cce@17.0.1 build_system=autotools patches=bbf97f1 arch=linux-sles15-zen2

Notice the [e]  annotation, this signifies an external dependency and that the system cray-mpich package will be used by Spack.

  1. install your specs:

    > spack install