intro_pgas

Date:: 07-30-2026

NAME

intro_pgas, pgas, coarray, intro_coarray - Introduce Coarray Fortran Partitioned Global Address Space (PGAS) programming

DESCRIPTION

Cray supports the Fortran coarray Partitioned Global Address Space (PGAS) programming model. Coarrays were adopted by the Fortran 2008 standard.

A PGAS application passes data between cooperating parallel processes. These parallel processes are called images in Fortran. For consistency, this man page uses the term processing element (PE). The PE number is one less than the Fortran image number; thus PE 0 refers to Fortran image 1.

Unlike MPI and SHMEM, the PGAS programming model primarily provides a programming language rather than programming library approach to writing Single Program Multiple Data (SPMD) applications. Communication between PEs is performed using assignment statements where the variables on either side have affinity to different PEs.

In the PGAS model, memory accessible to all PEs is said to be remotely accessible. This memory is partitioned such that each partition has affinity to a particular PE. A PE that references its local partition, the one having affinity to itself, executes a traditional load or store instruction. A PE that references a remote partition, one with affinity to a different PE, requires network communication if the two PEs are executing on different nodes. Within a node, communication occurs via shared memory.

Compiling and Linking

The -hcaf option is required to enable use of Fortran’s PGAS functionality on some platforms to avoid incompatibilities or resource contention with commonly used third-party libraries like OpenMPI and MVAPICH.

Note

The Coarray Fortran runtime depends on the Cray OpenSHMEMX library (libsma), which in turn depends on network transport libraries. These dependencies must be configured by the user. See the EXTERNAL DEPENDENCIES section of crayftn(1) for details.

For code samples refer to EXAMPLES. The following command creates an executable file:

% crayftn hello.f90 -o hello

For information about linking PGAS applications to use huge pages, see the intro_hugepages(1) man page.

Launching a PGAS Application

The examples below use the Application Level Placement Scheduler (APLS) launcher used on legacy Cray platforms. Other launchers (eg, SLURM) are supported. Please refer to your platform’s documentation for details on how to launch jobs on compute nodes.

The Application Level Placement Scheduler (ALPS) can launch the executable. Launch the PGAS application using 128 PEs:

% aprun -n 128 ./hello

To reserve a specific amount of symmetric heap space, set the XT_SYMMETRIC_HEAP_SIZE environment variable to the desired number of bytes. The suffixes k, K, m, M, g, and G are permitted to simplify requests for large values:

% export XT_SYMMETRIC_HEAP_SIZE=512M
% aprun -n 128 ./hello

The aprun syntax for launching Multiple Program, Multiple Data (MPMD) applications can be overloaded, by setting the PMI_JOB_IS_SPMD environment variable, to give certain PEs exclusive access to a node:

% export PMI_JOB_IS_SPMD
% aprun -n 1 -N 1 ./hello : -n 127 ./hello

This syntax is useful when some PEs require more memory than others. See the aprun(1) man page for more information about launching applications.

Mixing PGAS with MPI

Fortran coarrays are compatible with MPI. The programmer is responsible for calling MPI_Init(3) and MPI_Finalize(3) from their main program. The PGAS runtime automatically initializes itself prior to execution of the main program and finalizes itself upon completion of the main program, so unlike MPI there is no need to explicitly call an initialization or finalization function for PGAS. Note that deadlock scenarios are possible if care isn’t taken to ensure forward progress of both the PGAS runtime and MPI.

CCE’s PGAS runtime is designed to work with the HPE Cray MPI implementation. It may work with other third-party MPI implementations, but this is neither tested nor supported. PGAS support in the compiler is disabled by default on HPE platforms where third-party MPI libraries are used.

For more information about using MPI, refer to the mpi(3) man page.

FORTRAN FEATURES

This section lists some commonly used features. For a complete explanation of Fortran syntax and semantics, refer to the Fortran standard, ISO/IEC 1539-1:2010.

this_image(): Evaluates to an image number in the range 1 to num_images(), inclusive.
num_images(): Evaluates to the number of PEs in the job, equal to the aprun -n value.
lock(), unlock(): Lock synchronization.
sync all: Image control statement. Global barrier.
sync images: Image control statement. Selective synchronization.
sync memory: Image control statement. Fence to ensure that changes to remotely accessible memory are visible to other images.
critical, end critical: Begin and end a mutually exclusive region.

FORTRAN EXTENSIONS

Cray supports PGAS extensions to Fortran that are not part of the Fortran standard. Refer to the man pages for more information.

CO_SUM(): Collective sum.
CO_BCAST(): Collective broadcast.
CO_MIN(), CO_MAX(): Collective minimum and maximum.

EXAMPLES

Example 1: Fortran (hello.f90)

program main
    print *,"Hello from Fortran image",this_image(),"of",num_images()
end program

Cray Relaxed Ordering Extensions

There are two compiler directives available to users that can relax ordering of references to improve performance under certain circumstances.

The first directive is pgas defer_sync. Details are available in the defer_sync(7) manpage. Normally, the compiler must guard against references that could violate program ordering semantics by synchronizing the references. When a user knows that a reference will not target overlapping memory from a past or future reference between fences/barriers, a pgas defer_sync directive can be used to force synchronization to be delayed until the next fence instruction.

The second directive is pgas buffered_async. Details are available in the buffered_async(7) manpage. This can be used in place of a defer_sync directive to batch small references into bulk data transfers. No ordering or correctness guarantees are made between BA and non-BA references. No progress guarantees are made unless both the source and target are actively making BA references or are inside a barrier/fence. The total size of local buffers used for BA operations is controlled via the PGAS_BA_BUF_SIZE environment variable described below.

ENVIRONMENT VARIABLES

The following environment variables affect PGAS behavior.

XT_SYMMETRIC_HEAP_SIZE: Controls the size (in bytes) of the symmetric heap. The value set in this environment variable is interpreted as a number of bytes, unless the number is followed by a char that acts as a multiplier, where: g or G multiplies by 2**30 (gigabytes), k or K multiplies by 2**10 (kilobytes), m or M multiplies by 2**20 (megabytes). For example, the string 20m returns the integer value 20*2**20, or 20 megabytes. Only one multiplier is recognized, so 20kk will not produce the same value as 20m, nor will invalid strings such as 20MB produce the desired result.

Default: 64M

PGAS_ERROR_FILE [stderr | stdout]: To redirect error messages issued by the PGAS library (libpgas) to stdout, set this variable to stdout.

Default: stderr

PGAS_MEMINFO_DISPLAY: To examine memory usage in a coarray Fortran application, set this variable to a value of 1 or greater. This causes libpgas to display information about the job’s memory allocation during initialization.
PMI_JOB_IS_SPMD: If set, PMI treats a job launched via the ALPS MPMD format as an SPMD job instead. This enables PGAS jobs launched using the syntax

aprun -n X ./a.out : -n Y ./a.out

to run on Cray systems. This type of launch may be desired in order to assign extra resources such as memory to one or more PEs in the job.

Default: Not enabled

CRAY_PGAS_MAX_CONCURRENCY: By default, the PGAS runtime uses locks to ensure thread-safety if the number of PEs on a node is less than the number of available schedulable CPUs (including hyper-threads). In some cases, this can cause a noticeable overhead if the application does not actually need it (no OpenMP, no POSIX threads, etc). Setting this to 1 may improve performance in such cases. Setting this to a value greater than 1 may be necessary if the schedulable CPUs (including hyper-threads) is less than the total number of threads on node.

Default: unset (autodetected)

CRAY_PGAS_XPMEM_LIMIT: Controls the size (in bytes) of the shared memory segments that map memory from PEs located on the same compute node. The value set in this environment variable is interpreted as a number of bytes, unless the number is followed by a char that acts as a multiplier, where: g or G multiplies by 2**30 (gigabytes), k or K multiplies by 2**10 (kilobytes), m or M multiplies by 2**20 (megabytes). Setting this value too small may result in undefined behavior if any PE addresses local memory outside of this limit. Only set this environment variable if insufficient virtual address space errors occur at runtime. Use care that takes into account application behavior when choosing a value.

Default: unset (use default size limits)

CRAY_PGAS_XPMEM_MAP_IMMEDIATE: If set, force shared memory segments between PEs on the same compute node to be created upfront at application launch instead of being deferred until actual use. Use of this environment variable may result in insufficient virtual address space errors. This environment variable should only be set if it is not viable to amortize the costs of attaching shared memory segments during runtime.

Default: unset (map shared memory segments only when targeted)

CRAY_PGAS_BA_BUF_SIZE: Controls the size (in bytes) of the amount of memory used for buffers on each node. The value set in this environment variable is interpreted in the same way as XT_SYMMETRIC_HEAP_SIZE described above.

Note: This only applies to applications that have the buffered_async directive applied to references. By default, no buffer space is allocated.
Default: unset (defaults to none, or reserve a maximum of half the symmetric heap size when the pgas buffered_async directive is used)

NOTES

Supported number of PEs on a node

The application may encounter runtime errors due to lack of resources, including virtual address space, if the number of PEs exceeds the number of physical cores on the node. Some of these resource limits may be worked around to increase the maximum number of PEs per node if these errors are encountered. For example, hard ulimits for the data and stack segments can be lowered or the environment variable CRAY_PGAS_XPMEM_LIMIT set to the maximum amount of memory any single PE in the application uses (if this value is known) to reduce the runtime library’s use of virtual address space.