memory

Date:

04-16-2020

NAME

memory - Directive to place variables and memory allocations in specific types of memory

NOTICE

The Cray memory directive is deprecated and support will be discontinued in a future release.

Users are encouraged to prepare to transition to OpenMP 5.0 allocators instead, which will provide similar capabilities through standard mechanisms. CCE currently provides partial functional support for OpenMP 5.0 allocators, but allocator traits are currently ignored. Full support for mapping allocator traits is planned for a future CCE release.

SYNOPSIS

Fortran

!dir$ memory (list-of-attributes) [if(if-expr)]
heap-allocation-statement

!dir$ memory (list-of-attributes) [if(if-expr) ::] list-of-vars
variable-declaration

component-declaration
!dir$ memory (list-of-attributes) [if(if-expr) ::] list-of-components

IMPLEMENTATION

Cray Linux Environment (CLE)

DESCRIPTION

The memory directive is a mechanism that allows developers to place variables and memory allocations in a specific type of memory. This directive is intended for systems with more than one type of explicitly-addressable memory, particularly when the entire application footprint will not fit into the desired type of memory (or the developer would like to place different memory regions in different types of memory). For example, the Intel Xeon Phi (codenamed Knights Landing or KNL) has both “normal” DDR memory and a limited capacity of “high-bandwidth” MCDRAM memory.

The memory directive has two forms: one that applies to heap-allocation statements and one that applies to variable declarations.

Although this directive is most useful for systems with multiple types of memory, it is portable across all supported Cray systems, even ones that only have a single type of memory. CCE will always allocate memory that “best matches” the requested memory. Given this, the directive is not really useful for applications that are intended to only run on systems with a single type of memory. It is also not the most convenient solution for applications that can fit entirely within the desired type of memory, even if that application will run on a system with multiple types of memory. For this use case, the numactl command can be used to cause all memory to be allocated in the desired type of memory.

Note that this directive is only applicable to explicitly addressable memory. For the Intel Xeon Phi (KNL) processor, the directive cannot manage MCDRAM when it is configured as automatic cache mode.

Heap Allocations

!dir$ memory (list-of-attributes) [if(if-expr)]
heap-allocation-statement

This form of the directive appears before a heap-allocation statement in the base-language, causing the statement to allocate the specified type of memory rather than “normal” memory. list-of-attributes is a comma-separated list of attributes describing the type of memory that should be allocated. CCE currently supports three attributes, as follows:

bandwidth

This attribute maps to high-bandwidth MCDRAM on Intel Xeon Phi (KNL); for all other systems it maps to normal DDR.

capacity

This attribute maps to high-capacity memory. On Intel Xeon Phi (KNL) processors with MCDRAM, this attribute maps to normal DDR memory (which has a higher capacity than MCDRAM); for all other systems it maps to normal DDR.

fallback

This attribute specifies whether the allocation is allowed to automatically return to normal memory if it cannot be satisfied with the requested type of memory. Without the fallback clause (the default behavior), an allocation that cannot be satisfied with the requested type of memory will fail. Programmers may handle this failure through the normal mechanisms for failed allocations. The fallback attribute is ignored for systems with only a single type of memory, since there is nothing to fallback to.

The heap-allocation statement may be an allocate or deallocate statement.

The optional “if” clause provides dynamic control over the directive. It accepts a boolean or logical expression in the base language. If the expression evaluates to false at runtime then the directive is ignored; otherwise the directive behaves as specified.

Variable Allocations

!dir$ memory (list-of-attributes) [if(if-expr) ::] list-of-vars
variable-declaration

component-declaration
!dir$ memory (list-of-attributes) [if(if-expr) ::] list-of-components

This form of the directive applies to a variable declaration in the base language. The directive name is the same as for heap allocations, and it accepts the same list-of-attributes allowed for heap allocations. However, it also accepts a list of variable names that should be allocated in a particular type of memory. When applied to a variable declaration, the directive must appear before any executable statements. When applied to a component declaration, it must appear in the derived type declaration after the component declaration.

The list-of-vars is a comma-separated list of variable names. The variables may be local or global scope, including module variables. The variable may be any type, including scalars, fixed-size arrays, variable-length arrays, and derived types. The type may also be allocatable, including allocatable components of derived types; but, in this case the directive applies to the allocated memory rather than the underlying dope-vector. That is, the directive will apply only when the specified variable or component appears in an allocate statements. The directive may not be applied to derived-type components that are not allocatable.

The list-of-vars may not include:

  • Dummy arguments

  • Common blocks or variables within a common block

  • Fortran pointers

  • Variables involved in equivalences

  • Coarray variables

Just like for heap allocations, the optional “if” clause provides dynamic control over the directive. For variable declarations, the if-expr expression is evaluated when the variable goes into scope. For global variables and derived-type components if-expr must be a compile-time constant.

Linking

The standard CCE craylibs libraries provide runtime support for the base functionality of the memory directive. These libraries are automatically linked by the CCE compiler driver.

Support for MCDRAM on Intel Xeon Phi (KNL) processors is provided by the Intel memkind library, libmemkind. Loading the cray-memkind module will ensure that libmemkind is linked with the application. The memory directive may be used without linking against the libmemkind library, but if libmemkind cannot be located at runtime then the application will not be able to allocate MCDRAM (i.e., the system will be treated as though it only has normal DDR memory). The libmemkind library currently requires dynamic linking.

Huge Pages

Variables allocated with the memory directive will be allocated with huge pages if a craype_hugepages module is loaded or the HUGETLB_DEFAULT_PAGE_SIZE environment variable is set. Only 2M and 1G huge page sizes are supported for MCDRAM allocations, but for normal allocations all craype_hugepages modules are supported.

ENVIRONMENT VARIABLES

CRAYMEM_MEMKIND_PATH

May be used to specify the path to the memkind library, libmemkind.so; this environment variable is only needed if the memkind library is not available in the default search path.

CRAYMEM_DEBUG

May be used to enable runtime debug messages for the memory directive; the default value of 0 disables messages, while a positive value will specify the message level.

HUGETLB_DEFAULT_PAGE_SIZE

May be used to control the huge page size for variables allocated with the memory directive. This environment variable is automatically set by the craype_hugepages modules, but can be overridden explicitly. Only 2M and 1G huge pages are currently supported for MCDRAM allocations.

EXAMPLES

!dir$ memory(bandwidth)
allocate(var(1:n))
...
!dir$ memory(bandwidth) ! optional
deallocate(var)


real :: a(n), b(100), c
!dir$ memory(bandwidth) a, b, c


real,allocatable :: d(:) ! DV is in normal memory
!dir$ memory(bandwidth) d

type foo
  real,allocatable :: z(:)
  !dir$ memory(bandwidth) z
end type foo
type(foo) :: e
...
allocate (d(100), e%z(100)) ! allocates in HBM