MPT - DSMML
Overview
DSMML - Distributed Symmetric Memory Management Library
Implementation
Cray Linux Environment (CLE)
Introduction
DSMML is a stand-alone memory management library for maintaining distributed shared symmetric memory heaps for top level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. Through DSMML, we could extract interoperability between PGAS programming models.
Compiling and Launching a DSMML application on Cray system
To invoke the compiler for all applications, including standalone DSMML applications, use either the cc, CC, or ftn command. Do not use any vendor-specific compiler commands such as pgcc, as this may result in undefined behavior.
Example:
In the example below, an application is first compiled, and the resulting executable is then launched using 1 process:
cc -o test_dsmml test_dsmml.c
srun -n 1 ./test_dsmml
See srun(1) man page for more information.
Environment Variables
The following environment variables affect the DSMML behavior.
DSMML_VERSION_INFO
If set, causes DSMML to display the Cray DSMML library version number as
well as the build date information.
Default: not enabled
DSMML_ERROR_FILE
Specifies the name of error file to be used for reporting the errors in
DSMML. Only ```stderr``` and ```stdout``` are available as valid input
options.
Default: stderr
DSMML_DEBUG_FILE
Specifies the name of the file to be used for reporting the debug logs
in DSMML. Only ```stderr``` and ```stdout``` are available as valid
input options.
Default: stderr
DSMML_DEBUG_LEVEL
Specifies the level for providing the debug log details. The following
levels are available as valid input options, and their corresponding
information level details are provided.
Level:1 Only error reports
Level:2 General high-level user-facing function entry information
Level:3 High-level function entry information on all other functions
Level:4 More details on each internal operation
Level:5 Complete every possible log information
Default: 0
DSMML_DEBUG_RANKS
Specifies the ranks on which the debug logs are to be retrieved. Multiple
ranks can be combined using the (;) delimiter.
Default: (All ranks are reported)
DSMML_DEBUG_CATEGORIES
Debug categories used to isolate and separate the log information. The
following categories are available as input. Multiple categories can be
combined using the (;) delimiter.
init
enter
create
allocate
query
numa
Default: (All categories are reported)
Bug Report
Any DSMML specific error, information, and feature JIRA requests are to be
created with the category DSMML and under the DSMML PE-MPT project.
Summary
Create and initialize symmetric heap segment
Synopsis
dsmml_return_t dsmml_create_sheap_seg(dsmml_sheap_seg_info_t *seg_attrs);
Arguments
seg_attrs input segment information attributes
Description
Create amd initialize the symmetric heap segment based on the user input
arguments. Users are expected to pass various memory information like the
base address (virtual address space), length of the segment, memory type,
pagesize, and mode information throught the dsmml_sheap_seg_info_t(3).
Information on using the combination of the different types of memory and
mode of allocation.
Memory Types:
DSMML_MEM_SYS_DEFAULT
DSMML_MEM_NORMAL
DSMML_MEM_FAST
Different Modes:
DSMML_MODE_DEFAULT
DSMML_MODE_PREFERRED
DSMML_MODE_BIND
DSMML_MODE_INTERLEAVE
Effect of using different combinations of memory types and modes of allocation:
DSMML_MEM_SYS_DEFAULT DSMML_MEM_NORMAL DSMML_MEM_FAST
DSMML_MODE_DEFAULT NO EFFECT NO EFFECT NO EFFECT
DSMML_MODE_PREFERRED NO EFFECT DDR + MPOL_PREF HBM + MPOL_PREF
DSMML_MODE_BIND NO EFFECT DDR + MPOL_BIND HBM + MPOL_BIND
DSMML_MODE_INTERLEAVE NO EFFECT MPOL_INTERLEAVE MPOL_INTERLEAVE
Return Values
DSMML_RC_SUCCESS segment successfully created
DSMML_RC_INVALID_PARAM invalid parameter like NULL pointer passed
DSMML_RC_RESOURCE_ERROR unavailable memkind or hugepage size requested
DSMML_RC_NO_MEMORY heap memory cannot be mapped or heap too huge
DSMML_RC_UNKNOWN_FAIL unknown error
DSMML_RC_MEM_CORRUPT saw memory corruption on segment during alloc
Notes
Refer dsmml_sheap_seg_info_t(3)
Summary
Create shared symmetric heap segment
Synopsis
dsmml_return_t dsmml_create_ssheap_seg(dsmml_sheap_seg_info_t *seg_attrs);
Arguments
seg_attrs input segment information attributes
Description
Creates shared symmetric heap segment based on the segment information
attributes passed. While the shared symmetric heap segment is created
after returning from this routine, to utilize this shared symmetric heap
segment, users are required to share and initialize the segment using
dsmml_share_ssheap_seg(3) and dsmml_init_ssheap_seg(3).
Shared symmetric heap segments are usable only after being shared and
initialized.
Return Values
DSMML_RC_SUCCESS - segment successfully initialized
DSMML_RC_INVALID_PARAM - invalid parameter like NULL pointer passed
DSMML_RC_RESOURCE_ERROR - unavailable memkind or hugepage size requested
DSMML_RC_NO_MEMORY - heap memory cannot be mapped or heap too huge
DSMML_RC_UNKNOWN_FAIL - unknown error
DSMML_RC_MEM_CORRUPT - saw memory corruption on segment during alloc
DSMML_RC_UNKNOWN_FAIL - unknown error
Notes
Refer dsmml_share_ssheap_seg(3), dsmml_init_ssheap_seg(3)
Summary
Finalize and destroy all the segments created if the number of
initializations is matched.
Synopsis
dsmml_return_t dsmml_finalize(void);
Arguments
None.
Description
Finalize function, used to destroy all created segments in the DSMML
library. Operation is performed only when the number of initializations
matches the finalize calls.
Return Values
DSMML_RC_SUCCESS either all segments destroyed or init count reduced
DSMML_RC_FAILURE no initialization previously found
Notes
None.
Summary
Retrieve segment information using segment_id as input
Synopsis
dsmml_return_t dsmml_get_sheap_seg(int segment_id, dsmml_sheap_seg_info_t **seg_info_attr);
Arguments
segment_id segment id passed as input
seg_info_attrs output argument to retrieve the segment information
Description
dsmml_get_sheap_seg is used to receive segment information incase the top
level library requires it after the segment creation. Clients have to
provide the segment id as input to retrieve segment information
Return Values
DSMML_RC_SUCCESS segment info successfully retrieved
DSMML_RC_INVALID_PARAM invalid segment ID passed or op_seg not NULL
DSMML_RC_RESOURCE_ERROR no segment is created with this ID or no
segments are created before
Notes
Refer dsmml_sheap_seg_info_t(3)
Summary
Retrieve information of all the created segments are linked list
Synopsis
dsmml_return_t dsmml_get_sheap_seg_list(dsmml_sheap_seg_list_t **seg);
Arguments
seg output argument to retrieve the linked list of all created segments
Description
Receive a linked list of all the available segments incase the top level
library requires after creating the segment. This will return the starting
element of the list and applications
Return Values
DSMML_RC_SUCCESS actual list pointer retrieved
DSMML_RC_INVALID_PARAM ip_seg passed is not NULL
DSMML_RC_RESOURCE_ERROR no segments are created before
Notes
Refer dsmml_sheap_seg_list_t(3)
Summary
Retrieve DSMML major and minor version information
Synopsis
dsmml_return_t dsmml_get_version_info(int *major, int *minor);
Arguments
major major version information returned
minor minor version information returned
Description
dsmml_get_version_info returns the major and minor version information
from the DSMML library. This is a thread-safe query operation and it can
used before the DSMML library initialization.
Return Values
DSMML_RC_SUCCESS successfully retrieved version information
DSMML_RC_FAILURE Unable to retrive version information
Summary
Initialize the segment which was previously created as a shared symmetric
heap segment and share with other processes
Synopsis
dsmml_return_t dsmml_init_ssheap_seg(int segment_id);
Arguments
segment_id segment id
Description
Initializes the segment which was previously created as a shared symmetric
heap segment using the dsmml_create_ssheap_seg(3) operation and shared with
other processes using dsmml_share_ssheap_seg(3) operation. To process
this operation, all clients has to make sure that the segment is created
in all processes and are shared between them. Hence, a barrier-like
operation is required before calling this routine.
Shared symmetric heap segments are usable only after being shared and
initialized.
Return Values
DSMML_RC_SUCCESS - segment successfully initialized
DSMML_RC_INVALID_PARAM - invalid parameter like NULL pointer passed
DSMML_RC_RESOURCE_ERROR - unavailable memkind or hugepage size requested
DSMML_RC_NO_MEMORY - heap memory cannot be mapped or heap too huge
DSMML_RC_UNKNOWN_FAIL - unknown error
Notes
Refer dsmml_create_ssheap_seg(3), dsmml_share_ssheap_seg(3)
Summary
Initialization function to initialize the library and return either success
or failure based on the input argument provided.
Synopsis
dsmml_return_t dsmml_init(dsmml_init_info_t *init_attrs);
Arguments
init_attrs dsmml_init_info_t structure with all the initialization
attributes
Description
Initialization function to initialize the library. Clients are expected to
pass the setup arguments through dsmml_init_info_t(3) argument. Based on
the status of the initialization, library returns the return code.
Return Values
DSMML_RC_SUCCESS initialization successful
DSMML_RC_INVALID_PARAM invalid input argument parameters
DSMML_RC_NOOP if already initialized
Notes
Refer dsmml_init_info_t(3) manpage for more information about the input
arguments.
Multiple initialization and finalization operations are allowed.
Summary
DSMML structure for passing initialization input arguments into the library
Synopsis
typedef struct dsmml_init_info {
int mype;
int smp_mype;
int smp_npes;
} dsmml_init_info_t;
Members
mype my PE index information
smp_mype my local PE index
smp_npes total number of local PEs
Notes
Refer dsmml_init(3)
Summary
Shares the segment which was previously created as a shared symmetric heap
segment
Synopsis
dsmml_return_t dsmml_share_ssheap_seg(int segment_id);
Arguments
segment_id segment id
Description
Shares the segment which was previously created as a shared symmetric
heap segment using the dsmml_create_ssheap_seg(3) operation. To process
this operation, all clients has to make sure that the segment is created
in all processes where the segment is to be shared. Hence, a barrier-like
operation is required before calling this routine.
Shared symmetric heap segments are usable only after being shared and
initialized. It is initialized using dsmml_init_ssheap_seg(3).
Return Values
DSMML_RC_SUCCESS - segment successfully initialized
DSMML_RC_INVALID_PARAM - invalid parameter like NULL pointer passed
DSMML_RC_RESOURCE_ERROR - unavailable memkind or hugepage size requested
DSMML_RC_NO_MEMORY - heap memory cannot be mapped or heap too huge
DSMML_RC_UNKNOWN_FAIL - unknown error
Notes
Refer dsmml_create_ssheap_seg(3), dsmml_init_ssheap_seg(3)
Summary
Free previously created memory allocation
Synopsis
dsmml_return_t dsmml_sheap_free(int segment_id, void **ptr);
Arguments
segment_id segment id as input
ptr pointer to previously created memory allocation
Description
dsmml_sheap_free is ued to free previously created memory allocation
Return Values
DSMML_RC_SUCCESS memory successfully freed
DSMML_RC_INVALID_PARAM requested memory not previously allocated
DSMML_RC_RESOURCE_ERROR invalid segment ID passed
Notes
None.
Summary
Allocate memory from the given segment
Synopsis
dsmml_return_t dsmml_sheap_malloc(int segment_id, size_t length, void **ptr);
Arguments
segment_id segment id as input
length length of the memory to be allocated in bytes
ptr pointer to the allocated memory
Description
dsmml_sheap_malloc is used to allocate memory from previously created segment
Return Values
DSMML_RC_SUCCESS memory successfully allocated
DSMML_RC_INVALID_PARAM invalid param passed ptr != NULL, or segid neg
DSMML_RC_NO_MEMORY requested memory greater than available
DSMML_RC_RESOURCE_ERROR invalid segment ID passed
DSMML_RC_UNKNOWN_FAIL unknown error
DSMML_RC_MEM_CORRUPT saw memory corruption on segment during alloc
Notes
None.
Summary
Reallocate memory on the given segment
Synopsis
dsmml_return_t dsmml_sheap_realloc(int segment_id, void **alloc_ptr, size_t length,
void **realloc_ptr);
Arguments
segment_id segment as input
alloc_ptr pointer to the previously allocated memory
length length of the new reallocated memory
realloc_ptr output pointer to the reallocated memory
Description
Reallocate memory on a given segment over the already allocated memory.
Return Values
DSMML_RC_SUCCESS memory successfully reallocated
DSMML_RC_NO_MEMORY requested memory greater than available
DSMML_RC_RESOURCE_ERROR invalid segment ID passed
DSMML_RC_INVALID_PARAM invalid allocation paramter passed
Notes
None.
Summary
structure for creating and maintaining symmetric heap creation.
Synopsis
typedef struct dsmml_sheap_seg_info {
int id;
void * act_addr;
void * device_addr;
void * base_addr;
size_t length;
dsmml_type_t type;
dsmml_hpsize_t pagesize;
dsmml_mode_t mode;
int smp_mype;
int smp_npes;
int smp_set;
dsmml_nnode_t nnode;
int use_ext_mem;
} dsmml_sheap_seg_info_t;
Members
id output segment id
act_addr actual address for all future use
device_addr GPU shadow memory
base_addr base vaddr
length length of the segment in bytes
type memory type
pagesize hugepage size
mode allocation mode
smp_mype my load PE index on the SMP group
smp_npes total number of local PEs in my SMP group
smp_set SMP group index
nnode NUMA node details
use_ext_mem use external block memory management
Constants used for Memory Types:
DSMML_MEM_SYS_DEFAULT
DSMML_MEM_NORMAL
DSMML_MEM_FAST
Constants used for Different Modes:
DSMML_MODE_DEFAULT
DSMML_MODE_PREFERRED
DSMML_MODE_BIND
DSMML_MODE_INTERLEAVE
Constants used for Different HugePage Size supported:
DSMML_HPSIZE_DEFAULT - Default hugpage - mostly try using transparent HP
DSMML_HPSIZE_4K - System default pagesize
DSMML_HPSIZE_2M
DSMML_HPSIZE_4M
DSMML_HPSIZE_8M
DSMML_HPSIZE_16M
DSMML_HPSIZE_32M
DSMML_HPSIZE_64M
DSMML_HPSIZE_128M
DSMML_HPSIZE_256M
DSMML_HPSIZE_512M
DSMML_HPSIZE_1G
DSMML_HPSIZE_2G
Effect of using different combinations of memory types and modes of allocation:
DSMML_MEM_SYS_DEFAULT DSMML_MEM_NORMAL DSMML_MEM_FAST
DSMML_MODE_DEFAULT NO EFFECT NO EFFECT NO EFFECT
DSMML_MODE_PREFERRED NO EFFECT DDR + MPOL_PREF HBM + MPOL_PREF
DSMML_MODE_BIND NO EFFECT DDR + MPOL_BIND HBM + MPOL_BIND
DSMML_MODE_INTERLEAVE NO EFFECT MPOL_INTERLEAVE MPOL_INTERLEAVE
External Memory Management:
By default, segments are managed with metadata stored as part of the
segment. Enabling the external memory management allows managing the
segment metadata independtly of the segment memory.
Notes
Refer dsmml_sheap_seg_list_t(3), dsmml_get_sheap_seg(3), and
dsmml_create_sheap_seg(3),
Summary
structure for getting all the created symmetric heaps as linked list
Synopsis
typedef struct dsmml_sheap_seg_list {
struct dsmml_sheap_seg_list *next;
dsmml_sheap_seg_info_t * segment;
} dsmml_sheap_seg_list_t;
Members
next next segment in the linked list
segment pointer with the segment information
Notes
Refer dsmml_sheap_seg_info_t(3), and dsmml_get_sheap_seg_list(3)