compiler_interop

Date:

07-31-2026

NAME

compiler_interop - Overview of CCE Fortran compiler interoperability

DESCRIPTION

This man page describes the Cray Fortran compiler’s cross-language and cross-vendor interoperability capabilities and limitations. Successfully supporting large HPC applications requires several different forms of compiler and programming language interoperability. Applications may require interoperability across two or more base languages (e.g., C, C++, or Fortran) and programming models (e.g., OpenMP, OpenACC, HIP, or CUDA). This often occurs when an application comprises several independent subcomponents, which may be developed by different teams. Additionally, applications may require interoperability across two or more compiler vendors.

Fortran Interoperability

The CCE Fortran compiler supports interoperability between Fortran and C through Fortran language standard C interoperability mechanisms. Cross-vendor Fortran interoperability is not generally supported because the Fortran language specification does not define a standard ABI or runtime library interface. All Fortran code contained in an application must typically be compiled by a single Fortran compiler.

GCC Header and Library Selection

The CCE Fortran compiler automatically uses required GCC libraries and searches the system for installed GCC libraries, selecting the highest GCC version found. Users may direct the compiler to use specific GCC libraries via the mechanisms described in this section.

Command Line Options

When the **–gcc-install-dir=***lib-dir* option is provided, the compiler uses the GCC libraries in lib-dir. For example, the following option directs the compiler to use the GCC libraries in /usr/lib64/gcc/x86_64-suse-linux/13:

--gcc-install-dir=/usr/lib64/gcc/x86_64-suse-linux/13

When the **–gcc-toolchain=***prefix* option is provided, the compiler uses the GCC libraries from the highest GCC version found in prefix. For example, the following option directs the compiler to use the highest GCC version found in /usr:

--gcc-toolchain=/usr

The –gcc-install-dir= option (with nothing after the =) sets lib-dir to the empty string. The –gcc-toolchain= option (with nothing after the =) sets prefix to the empty string.

Precedence Rules

When multiple **–gcc-install-dir=***lib-dir* options are provided, only the last lib-dir takes effect. When multiple **–gcc-toolchain=***prefix* options are provided, only the last prefix takes effect. A **–gcc-install-dir=***lib-dir* option on the command line overrides any lib-dir provided via a configuration file. A **–gcc-toolchain=***prefix* option on the command line overrides any prefix provided via a configuration file. These precedence rules apply equally to empty strings and nonempty strings.

The compiler considers all configuration files and command line options, and computes a final lib-dir and prefix. If lib-dir is nonempty, the compiler only uses lib-dir. If lib-dir is empty and prefix is nonempty, the compiler only uses prefix. If lib-dir is empty and prefix is empty, the compiler uses its default GCC search algorithm. As a result, a prefix provided on the command line does not override a nonempty lib-dir provided via a configuration file.

Clearing lib-dir and prefix directs the compiler to search the system for the highest GCC version installed.

Configuration Files

Administrators may set systemwide GCC selection defaults by creating and editing compiler configuration files.

The Fortran compiler reads two configuration files that affect GCC selection: gcc_install_dir and gcc_toolchain. These files are in the share directory adjacent to the bin directory that contains the ftnfe executable. For example, in CCE 18.0.0 on x86_64, the gcc_install_dir and gcc_toolchain files are in the following directory:

/opt/cray/pe/cce/18.0.0/cce/x86_64/bin/../share

The Fortran compiler interprets the first line of gcc_install_dir as the lib-dir for the –gcc-install-dir option. The Fortran compiler interprets the first line of gcc_toolchain as the prefix for the –gcc-toolchain option. Providing a nonempty gcc_install_dir or gcc_toolchain file directs the Fortran compiler to use the specified GCC selection option by default.

Debugging

When the CCE_DEBUG_GCC_SELECTION environment variable is set to 1, the Fortran compiler prints GCC selection messages to standard error (stderr).

Runtime Library Dependences

Object files produced by the CCE Fortran compiler can have references to symbols in a number of CCE compiler runtime libraries, and libraries and executable binaries will have dynamic library dependences on those runtime libraries. Some library dependences are always needed and others may be triggered based on compiler/linker flags.

Fortran binaries produced by CCE will depend on the following runtime libraries:

• libu - CCE compiler runtime

• libcsup - CCE compiler runtime

• libcraymath - CCE optimized math library

• libquadmath - CCE 128-bit floating-point math library

• libf - Fortran compiler runtime

• libfi - Fortran compiler runtime

• libmodules - Fortran standard modules

OpenMP and OpenACC binaries produced by CCE will depend on the following runtime libraries:

• libcraymp - OpenMP runtime library for CPU threading

• libcrayacc - OpenMP/OpenACC runtime library for NVIDIA GPU offloading

• libcrayacc_amdgpu - OpenMP/OpenACC runtime library for AMD GPU offloading

• libcrayacc_x86_64 - OpenMP/OpenACC runtime library for x86-64 host offloading

• libcrayacc_aarch64 - OpenMP/OpenACC runtime library for aarch64 host offloading

OpenMP Interoperability

CCE supports limited OpenMP interoperability between Cray Fortran and other GNU and Clang based compilers. The mechanisms that enable such interoperability are CCE’s OpenMP runtime libraries and custom offload linker tool.

CCE provides two OpenMP runtime libraries: libcraymp, which is the primary CPU OpenMP runtime library needed by all OpenMP programs; and libcrayacc, which is the offloading library that is only needed for GPU offloading.

While CCE internally uses a non-standard compiler-runtime interface, CCE’s OpenMP runtime libraries also implement entry points for the GNU and Clang compiler-runtime interfaces. As a result, libcraymp can act as a drop-in replacement for GNU’s libgomp runtime library and Clang’s libomp runtime library; and, libcrayacc can act as a drop-in replacement for Clang’s libomptarget runtime library. This allows compiling OpenMP source files with CCE and other GNU or Clang based compilers and then linking against the CCE OpenMP runtime libraries, enabling cross-vendor OpenMP interoperability within a single application. Constraints for cross-vendor OpenMP offloading are as follows:

• OpenMP interoperability is supported for Clang-based compilers when the LLVM version is less than or equal to CCE’s LLVM version; interoperability with objects compiled by newer versions of LLVM may work, but cannot be guaranteed. All OpenMP constructs and APIs are supported for Clang based compilers.

• OpenMP interoperability is supported for GNU-based compilers for all otherwise-supported OpenMP constructs and APIs with the exception of target offload constructs and APIs.

• OpenMP interoperability for GPU offloading is only supported for Clang-based compilers; only OpenMP CPU interoperability is supported for GNU-based compilers.

• GPU offloading interoperability does not support cross-vendor function calls or references to global variables (i.e., accesses to shared global variables or function calls in offload regions need to be contained in files compiled by the same compiler). An application can contain offload regions that are compiled by different compilers as long as those regions are independent.

• Applications must be linked against the CCE OpenMP runtime libraries, libcraymp and libcrayacc, rather than the GNU or Clang OpenMP runtime libraries. This is most easily accomplished by linking the application with the CCE compiler driver.

• Applications that contain GPU offloading regions must be linked with the CCE OpenMP Offload Linker tool, which handles all of the device linking steps needed for “bundled” offload objects. The offload linker tool is invoked automatically when CCE is used to perform a final link command. The offload linker tool can also be invoked directly in a pre-link step when linking an application with another compiler driver. See the CCE OpenMP Offload Linker section in the intro_openmp man page for more details.

CCE provides distinct libcrayacc library builds for each supported GPU target, and each of these libraries has a direct dependence on the offload libraries provided by the corresponding GPU vendor.

• libcrayacc is CCE’s NVIDIA offloading library, and it has a dependence on NVIDIA’s libcuda library.

• libcrayacc_amdgpu is CCE’s AMD GPU offloading library, and it has a dependence on AMD’s libamdhip64 library.

• libcrayacc_x86_64 and libcrayacc_aarch64 are CCE’s CPU offloading libraries; these libraries support “host offloading”, and therefore do not have a dependence on any third-party GPU offloading libraries.

For interoperability with native offloading models, such as CUDA or HIP, CCE supports the standard OpenMP interop construct and associated API functions. CCE does not generally support function calls from OpenMP offload kernels to CUDA/HIP device functions and from CUDA/HIP kernels to OpenMP offload device functions. However, additional CUDA cubin binary or PTX assembly files (when targeting NVIDIA GPUs) or additional device LLVM bitcode files (when targeting AMD GPUs) can be linked in with OpenMP offloading device code using the -Wc,<cubin> driver flag. This provides basic capability for invoking CUDA/HIP device functions from OpenMP/OpenACC kernels.

OpenACC Interoperability

The CCE Fortran OpenACC implementation uses the same offloading runtime library as OpenMP, libcrayacc, and therefore natively supports mixing OpenACC and OpenMP offload in a single application. CCE does not support OpenACC interoperability with other compilers.

HIP Interoperability

CCE supports interoperability for applications containing a mix of Cray Fortran and HIP object files. Cray Fortran source files, including those that use OpenACC or OpenMP for AMD GPU offloading, are compiled with CCE. HIP source files must be compiled with a HIP compiler, such as AMD’s HIP compiler or another HIP-enabled Clang compiler.

When HIP source files are compiled without relocatable device code, the resulting host object files contain executable HIP device code that does not require a final device linking step. These object files can be linked directly with crayftn. Linking with crayftn is recommended when the application also contains Cray Fortran OpenMP or OpenACC offload code, because crayftn is required to link the Cray Fortran device code.

HIP source files compiled with relocatable device code (-fgpu-rdc) require a final HIP device linking step, which normally requires using the HIP compiler for the final application link. Cray Fortran OpenMP or OpenACC offload code similarly requires using crayftn for the final application link unless its device code is pre-linked separately. An application containing both kinds of device code therefore has competing final-link requirements.

The recommended solution is to use crayftn for the final application link. crayftn can link HIP relocatable device code natively when it was produced by a HIP compiler using the same LLVM version as CCE. Specify the --hip-link and --offload-arch=ARCH options to request the HIP device link; crayftn also links the Cray Fortran offload code.

When the HIP compiler must perform the final application link, invoke the COOL tool directly in device-link-only mode on the Cray Fortran offload objects. Pass the resulting host object, the original application objects, and the required CCE compiler runtime libraries to the HIP compiler for the final link. This pre-link step removes the requirement to use crayftn as the final linker. See the CCE OpenMP Offload Linker section in the intro_openmp man page for details.

CUDA Interoperability

CCE supports interoperability for applications containing a mix of CUDA and non-CUDA sources. CCE can be used to compile Fortran source files, including those that use OpenACC and OpenMP for CPU threading and NVIDIA GPU offloading. For best results, CUDA source files should be compiled with NVIDIA’s nvcc CUDA compiler.

When CUDA source files are compiled without enabling relocatable device code (the default behavior for nvcc, equivalent to specifying the --relocatable-device-code=false flag), the resulting host object files will contain executable CUDA object code that does not require any final device linking steps. As a result, these object files can be linked directly with crayftn. Linking with crayftn is recommended when the application also contains Cray Fortran OpenMP or OpenACC offload code, because crayftn is required to link the Cray Fortran device code.

If any CUDA source files are compiled with relocatable device code (--relocatable-device-code=true), the resulting host object files will contain relocatable CUDA object code that requires a final device linking step, which normally requires using nvcc for the final application link. Cray Fortran OpenMP or OpenACC offload code similarly requires using crayftn for the final application link unless its device code is pre-linked separately. An application containing both kinds of device code therefore has competing final-link requirements.

The recommended solution is to use nvcc --device-link to perform only the CUDA device link. This produces a host object containing the linked CUDA device image. Pass that object and the original application objects to crayftn for the final application link; crayftn also links the Cray Fortran offload code.

Alternatively, when nvcc must perform the final application link, invoke the COOL tool directly in device-link-only mode on the Cray Fortran offload objects. Pass the resulting host object, the original application objects, and the required CCE compiler runtime libraries to nvcc for the final link. This pre-link step removes the requirement to use crayftn as the final linker. See the CCE OpenMP Offload Linker section in the intro_openmp man page for details.

Cray Programming Environment Implications

Building a single executable or shared library with multiple compilers requires invoking each compiler driver directly. CCE supports direct invocation of the Fortran compiler through the crayftn command. Users must explicitly specify the appropriate compiler targeting flags (e.g., -hcpu, -haccel, -hnetwork), include paths, and linker flags, including flags needed for MPI. Alternatively, Cray MPI provides a set of MPI compiler wrappers (mpicc, mpicxx, and mpifort) that add the compiler and linker flags needed for compiling with Cray MPI.

Example

This section provides a demonstration of heterogeneous compiler interoperability using the open-source “Goulash” project authored by developers at Lawrence Livermore National Laboratory (https://github.com/LLNL/goulash). This demonstration focuses on the “rush_larsen_interop2.5_gpu_omp_hip_mpi” test case, which exercises MPI, HIP GPU offload, OpenMP GPU offload, and OpenMP CPU threading from C++ and Fortran using several different compiler toolchains – AMD ROCm (C++), GNU (C++), and CCE (Fortran). The target hardware is an HPE Cray EX supercomputer containing compute nodes populated with AMD EPYC “Trento” CPUs and AMD MI250X GPUs. HIP code is compiled with relocatable device code (-fgpu-rdc) because that is consistent with how many HPC applications are built, even though it is not strictly required for this test case. The configure and build setup is as follows:

# Specify common compiler flags
export OMP_CPU_CXXFLAGS="-O3 -fopenmp"
export OMP_GPU_CXXFLAGS="-O3 -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a --rocm-path=${ROCM_PATH}"
export HIP_CPPFLAGS="-D__HIP_PLATFORM_AMD__ -I${ROCM_PATH}/include"
export HIP_CXXFLAGS="-O3 --offload-arch=gfx90a -std=c++11 --rocm-path=${ROCM_PATH} -x hip -fgpu-rdc"

# Setup MPI wrappers for each compiler vendor using  an "env" command
# to extend the the PATH environment variable, allowing the MPI wrapper
# to find the appropriate compiler driver
export CCE_MPIFORT="env MPICH_FC=$(which crayftn) $(which mpifort)"
export ROCM_MPICXX="env MPICH_CXX=$(which amdclang) $(which mpicxx)"
export GNU_MPICXX="env MPICH_CXX=$(which g++) $(which mpicxx)"

# Capture library search paths
export INTEROP_LD_LIBRARY_PATH=${ROCM_PATH}/lib:$(pkg-config libfabric --variable=libdir)

# Define variables consumed by Goulash makefile
export INTEROP_CXX="${ROCM_MPICXX}"
export INTEROP_CPPFLAGS="${HIP_CPPFLAGS}"
export INTEROP_CXXFLAGS="${OMP_GPU_CXXFLAGS}"
export INTEROP_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export INTEROP_LDFLAGS="--hip-link --offload-arch=gfx90a -L${ROCM_PATH}/lib"
export INTEROP_LDLIBS="-lamdhip64 -lmpifort_cray"
export COMPILER1_GPU_OMP_HIP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER1_GPU_OMP_HIP_MPI_CPPFLAGS="${HIP_CPPFLAGS}"
export COMPILER1_GPU_OMP_HIP_MPI_CXXFLAGS="${OMP_GPU_CXXFLAGS}"
export COMPILER1_GPU_OMP_HIP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER1_GPU_OMP_HIP_MPI_LDFLAGS="${INTEROP_LDFLAGS}"
export COMPILER1_GPU_OMP_HIP_MPI_LDLIBS="${INTEROP_LDLIBS}"
export COMPILER1_CPU_OMP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER1_CPU_OMP_MPI_CPPFLAGS=""
export COMPILER1_CPU_OMP_MPI_CXXFLAGS="${OMP_CPU_CXXFLAGS}"
export COMPILER1_CPU_OMP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER1_CPU_OMP_MPI_LDFLAGS=""
export COMPILER1_CPU_OMP_MPI_LDLIBS=""
export COMPILER2_CPU_OMP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER2_CPU_OMP_MPI_CPPFLAGS=""
export COMPILER2_CPU_OMP_MPI_CXXFLAGS="${OMP_CPU_CXXFLAGS}"
export COMPILER2_CPU_OMP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER2_CPU_OMP_MPI_LDFLAGS=""
export COMPILER2_CPU_OMP_MPI_LDLIBS=""
export COMPILER3_CPU_OMP_MPI_CXX="${GNU_MPICXX}"
export COMPILER3_CPU_OMP_MPI_CPPFLAGS=""
export COMPILER3_CPU_OMP_MPI_CXXFLAGS="${OMP_CPU_CXXFLAGS} -fPIC"
export COMPILER3_CPU_OMP_MPI_COMPILERID="-DCOMPILERID=\"$(${COMPILER3_CPU_OMP_MPI_CXX} --version | head -n1)\""
export COMPILER3_CPU_OMP_MPI_LDFLAGS=""
export COMPILER3_CPU_OMP_MPI_LDLIBS=""
export COMPILER1_GPU_OMP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER1_GPU_OMP_MPI_CPPFLAGS=""
export COMPILER1_GPU_OMP_MPI_CXXFLAGS="${OMP_GPU_CXXFLAGS}"
export COMPILER1_GPU_OMP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER1_GPU_OMP_MPI_LDFLAGS=""
export COMPILER1_GPU_OMP_MPI_LDLIBS=""
export COMPILER2_GPU_OMP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER2_GPU_OMP_MPI_CPPFLAGS=""
export COMPILER2_GPU_OMP_MPI_CXXFLAGS="${OMP_GPU_CXXFLAGS}"
export COMPILER2_GPU_OMP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER2_GPU_OMP_MPI_LDFLAGS=""
export COMPILER2_GPU_OMP_MPI_LDLIBS=""
export COMPILER1_GPU_HIP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER1_GPU_HIP_MPI_CPPFLAGS="${HIP_CPPFLAGS}"
export COMPILER1_GPU_HIP_MPI_CXXFLAGS="${HIP_CXXFLAGS}"
export COMPILER1_GPU_HIP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER1_GPU_HIP_MPI_LDFLAGS=""
export COMPILER1_GPU_HIP_MPI_LDLIBS=""
export COMPILER2_GPU_HIP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER2_GPU_HIP_MPI_CPPFLAGS="${HIP_CPPFLAGS}"
export COMPILER2_GPU_HIP_MPI_CXXFLAGS="${HIP_CXXFLAGS}"
export COMPILER2_GPU_HIP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER2_GPU_HIP_MPI_LDFLAGS=""
export COMPILER2_GPU_HIP_MPI_LDLIBS=""
export COMPILER1_GPU_LAMBDA_HIP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER1_GPU_LAMBDA_HIP_MPI_CPPFLAGS="${HIP_CPPFLAGS}"
export COMPILER1_GPU_LAMBDA_HIP_MPI_CXXFLAGS="${HIP_CXXFLAGS}"
export COMPILER1_GPU_LAMBDA_HIP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER1_GPU_LAMBDA_HIP_MPI_LDFLAGS=""
export COMPILER1_GPU_LAMBDA_HIP_MPI_LDLIBS=""
export COMPILER2_GPU_LAMBDA_HIP_MPI_CXX="${ROCM_MPICXX}"
export COMPILER2_GPU_LAMBDA_HIP_MPI_CPPFLAGS="${HIP_CPPFLAGS}"
export COMPILER2_GPU_LAMBDA_HIP_MPI_CXXFLAGS="${HIP_CXXFLAGS}"
export COMPILER2_GPU_LAMBDA_HIP_MPI_COMPILERID="-DCOMPILERID=$(basename ${ROCM_PATH})"
export COMPILER2_GPU_LAMBDA_HIP_MPI_LDFLAGS=""
export COMPILER2_GPU_LAMBDA_HIP_MPI_LDLIBS=""
export COMPILER1_CPU_OMP_MPI_FORT_FC="${CCE_MPIFORT}"
export COMPILER1_CPU_OMP_MPI_FORT_CPPFLAGS=""
export COMPILER1_CPU_OMP_MPI_FORT_FCFLAGS="${OMP_CPU_CXXFLAGS}"
export COMPILER1_CPU_OMP_MPI_FORT_COMPILERID="-DCOMPILERID=\\\"cce-${CRAY_FTN_VERSION}\\\""
export COMPILER1_CPU_OMP_MPI_FORT_LDFLAGS=""
export COMPILER1_CPU_OMP_MPI_FORT_LDLIBS=""
export COMPILER1_GPU_OMP_MPI_FORT_FC="${CCE_MPIFORT}"
export COMPILER1_GPU_OMP_MPI_FORT_CPPFLAGS=""
export COMPILER1_GPU_OMP_MPI_FORT_FCFLAGS="${OMP_CPU_CXXFLAGS} -haccel=amd_gfx90a -gno-heterogeneous-dwarf"
export COMPILER1_GPU_OMP_MPI_FORT_COMPILERID="-DCOMPILERID=\\\"cce-${CRAY_FTN_VERSION}\\\""
export COMPILER1_GPU_OMP_MPI_FORT_LDFLAGS=""
export COMPILER1_GPU_OMP_MPI_FORT_LDLIBS=""

# Purge modules after capturing all necessary paths
module purge

git clone https://github.com/LLNL/goulash

# Build test case
make -C goulash/tests/rush_larsen/rush_larsen_interop2.5_gpu_omp_hip_mpi -e

The individual compile and link commands from an example build are shown below:

env MPICH_FC=/opt/cray/pe/cce/17.0.1/bin/crayftn /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpifort \
    -c -DCOMPILERID=\"cce-17.0.1\" -O3 -fopenmp -haccel=amd_gfx90a -gno-heterogeneous-dwarf \
    rush_larsen_gpu_omp_mpi_fort_compiler1.F90 -o rush_larsen_gpu_omp_fort_compiler1_mpi.o

env MPICH_FC=/opt/cray/pe/cce/17.0.1/bin/crayftn /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpifort \
    -c -DCOMPILERID=\"cce-17.0.1\" -O3 -fopenmp \
    rush_larsen_cpu_omp_mpi_fort_compiler1.F90 -o rush_larsen_cpu_omp_fort_compiler1_mpi.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -D__HIP_PLATFORM_AMD__ -I/opt/rocm-6.0.0/include -O3 --offload-arch=gfx90a \
    -std=c++11 --rocm-path=/opt/rocm-6.0.0 -x hip -fgpu-rdc \
    rush_larsen_gpu_lambda_hip_mpi_compiler2.cc -o rush_larsen_gpu_lambda_hip_mpi_compiler2.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -D__HIP_PLATFORM_AMD__ -I/opt/rocm-6.0.0/include -O3 --offload-arch=gfx90a \
    -std=c++11 --rocm-path=/opt/rocm-6.0.0 -x hip -fgpu-rdc \
    rush_larsen_gpu_lambda_hip_mpi_compiler1.cc -o rush_larsen_gpu_lambda_hip_mpi_compiler1.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -D__HIP_PLATFORM_AMD__ -I/opt/rocm-6.0.0/include -O3 --offload-arch=gfx90a \
    -std=c++11 --rocm-path=/opt/rocm-6.0.0 -x hip -fgpu-rdc \
    rush_larsen_gpu_hip_mpi_compiler2.cc -o rush_larsen_gpu_hip_mpi_compiler2.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -D__HIP_PLATFORM_AMD__ -I/opt/rocm-6.0.0/include -O3 --offload-arch=gfx90a \
    -std=c++11 --rocm-path=/opt/rocm-6.0.0 -x hip -fgpu-rdc \
    rush_larsen_gpu_hip_mpi_compiler1.cc -o rush_larsen_gpu_hip_mpi_compiler1.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -O3 -fopenmp \
    -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a \
    --rocm-path=/opt/rocm-6.0.0 \
    rush_larsen_gpu_omp_mpi_compiler2.cc -o rush_larsen_gpu_omp_mpi_compiler2.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -O3 -fopenmp \
    -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a \
    --rocm-path=/opt/rocm-6.0.0 \
    rush_larsen_gpu_omp_mpi_compiler1.cc -o rush_larsen_gpu_omp_mpi_compiler1.o

env MPICH_CXX=/usr/bin/g++ /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID="g++ (SUSE Linux) 7.5.0" -O3 -fopenmp -fPIC \
    rush_larsen_cpu_omp_mpi_compiler3.cc -o rush_larsen_cpu_omp_mpi_compiler3.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -O3 -fopenmp \
    rush_larsen_cpu_omp_mpi_compiler2.cc -o rush_larsen_cpu_omp_mpi_compiler2.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -O3 -fopenmp \
    rush_larsen_cpu_omp_mpi_compiler1.cc -o rush_larsen_cpu_omp_mpi_compiler1.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -c -DCOMPILERID=rocm-6.0.0 -D__HIP_PLATFORM_AMD__ -I/opt/rocm-6.0.0/include -O3 -fopenmp \
    -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a \
    --rocm-path=/opt/rocm-6.0.0 \
    rush_larsen_interop2.5_gpu_omp_hip_mpi.cc -o rush_larsen_interop2.5_gpu_omp_hip_mpi.o

env MPICH_CXX=/opt/rocm-6.0.0/bin/amdclang /opt/cray/pe/mpich/8.1.29/ofi/crayclang/17.0/bin/mpicxx \
    -DCOMPILERID=rocm-6.0.0 -D__HIP_PLATFORM_AMD__ -I/opt/rocm-6.0.0/include -O3 -fopenmp \
    -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a \
    --rocm-path=/opt/rocm-6.0.0 \
    rush_larsen_gpu_omp_fort_compiler1_mpi.o rush_larsen_cpu_omp_fort_compiler1_mpi.o \
    rush_larsen_gpu_lambda_hip_mpi_compiler2.o rush_larsen_gpu_lambda_hip_mpi_compiler1.o \
    rush_larsen_gpu_hip_mpi_compiler2.o rush_larsen_gpu_hip_mpi_compiler1.o \
    rush_larsen_gpu_omp_mpi_compiler2.o rush_larsen_gpu_omp_mpi_compiler1.o \
    rush_larsen_cpu_omp_mpi_compiler3.o rush_larsen_cpu_omp_mpi_compiler2.o \
    rush_larsen_cpu_omp_mpi_compiler1.o rush_larsen_interop2.5_gpu_omp_hip_mpi.o \
    --hip-link --offload-arch=gfx90a -L/opt/rocm-6.0.0/lib -lamdhip64 -lmpifort_cray \
    -o rush_larsen_interop2.5_gpu_omp_hip_mpi

The resulting executable was launched with two MPI ranks across two compute nodes using 32 OpenMP threads each, performing 10 iterations of each variant, each performing 10 trials with a 10 GB data size:

# Run test case
env LD_LIBRARY_PATH=${INTEROP_LD_LIBRARY_PATH} \
    OMP_NUM_THREADS=32 \
    ${SRUN} -n 2 -N 2 -c 32 \
    goulash/tests/rush_larsen/rush_larsen_interop2.5_gpu_omp_hip_mpi/rush_larsen_interop2.5_gpu_omp_hip_mpi 10 10 10

The executable emits frequent progress messages as it runs, indicating that the internal validation checks have passed at each step. The data checks for the last iteration appear as follows:

0:  41.113 (0.416s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 cpu_omp_mpi_compiler1 [rocm-6.0.0]
0:  41.197 (0.427s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 cpu_omp_mpi_compiler2 [rocm-6.0.0]
0:  45.830 (0.418s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 cpu_omp_mpi_compiler3 [g++ (SUSE Linux) 7.5.0]
0:   1.767 (0.409s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 gpu_omp_mpi_compiler1 [rocm-6.0.0]
0:   1.964 (0.409s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 gpu_omp_mpi_compiler2 [rocm-6.0.0]
0:   1.771 (0.406s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 gpu_hip_mpi_compiler1 [rocm-6.0.0]
0:   1.779 (0.408s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 gpu_hip_mpi_compiler2 [rocm-6.0.0]
0:   1.767 (0.408s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 gpu_lambda_hip_mpi_compiler1 [rocm-6.0.0]
0:   1.769 (0.407s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 gpu_lambda_hip_mpi_compiler2 [rocm-6.0.0]
0:  23.648 (0.808s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 cpu_omp_mpi_fort_compiler1 [cce-17.0.1]
0:   2.140 (0.394s): PASSED Data check 10 10.00000000  m_gate[0]=0.976324219401755 gpu_omp_mpi_fort_compiler1 [cce-17.0.1]

The final output messages indicate that the test completed successfully:

IOP Rank 0: 1593.631 (2.162s): Measuring free GPU memory (all tasks) after calling rush_larsen_gpu_omp_mpi_fort_compiler1()  Iter 10 of 10
IOP Rank 0: 1593.645 (0.013s): VARGPUMEM:    0.00% GPU mem variation after calling rush_larsen_gpu_omp_mpi_fort_compiler1()  Iter 10 of 10
IOP Rank 0: 1593.645 (0.000s): MINGPUMEM:   53.77 gb free GPU memory after calling rush_larsen_gpu_omp_mpi_fort_compiler1()  Iter 10 of 10
IOP Rank 0: 1593.645 (0.000s): AVGGPUMEM:   53.77 gb free GPU memory after calling rush_larsen_gpu_omp_mpi_fort_compiler1()  Iter 10 of 10
IOP Rank 0: 1593.645 (0.000s): MAXGPUMEM:   53.77 gb free GPU memory after calling rush_larsen_gpu_omp_mpi_fort_compiler1()  Iter 10 of 10
IOP Rank 0: 1593.645 (0.000s): GPUMEMFREE:  53.77 gb free GPU memory after calling rush_larsen_gpu_omp_mpi_fort_compiler1()  Iter 10 of 10
IOP Rank 0: 1593.645 (0.000s): INTEROP_PASS: ========== Completed interoperability tests 10 10 10.00000000 (interop_gpu_omp_hip_mpi [rocm-6.0.0]) ==========

SEE ALSO

crayftn(1), intro_openmp(7)