grid_order

MPI rank order list generator

Author:: Hewlett Packard Enterprise Development LP.
Copyright:: Copyright 2019,2021-2024 Hewlett Packard Enterprise Development LP.
Manual section:: 1

SYNOPSIS

grid_order -C|-R [-P|-Z|-H] -g N1,N2,… -c n1,n2,… [-o d1,d2,…] [-m max] [-N ranks_per_node] [-n ranks_per_line] [-q] [-s starting_rank] [-T] [i1,i2,…]

DESCRIPTION

The grid_order utility is used to generate a rank order list for use by an MPI application that uses communication between nearest neighbors in a grid. When executed with the desired arguments, grid_order generates rank order information in the appropriate format and writes it to stdout. This output can then be copied or written into a file named MPICH_RANK_ORDER and used with the MPICH_RANK_REORDER_METHOD environment variable to override the default MPI rank placement scheme and specify a custom rank placement.

Note: For more information about using custom rank placement files and MPICH_RANK_REORDER_METHOD 3, see the intro_mpi(3) man page.

OPTIONS

The grid_order utility supports the following arguments:

-C | -R: Specify either -C for column-major or -R for row-major numbering.
[-P | -Z | -H]: Optionally, specify the order of the dimensions along which successive cells are filled. The -Z option (default) lists successive rows of cells in the same order, jumping from the end of one row to the beginning of the next. The -P option (Peano) lists successive rows in alternating order. The -H option (Hilbert) lists ranks in Hilbert curve order.
-g N1,N2,…: Grid dimensions.
-c n1,n2,…: Dimensions of the cells into which the grid is subdivided. All rank numbers that fall into a given cell will be listed consecutively in the rank order that is produced. It is preferable, but not required, that each cell dimension evenly divides the corresponding grid dimension, and that the product of the cell dimensions is equal to the number ranks that will be placed on a node. For Hilbert orders, this option need not be specified and is overridden if it is specified.
[-o d1,d2,…]: Order of ranks within cells and cells within the grid.
[-m max]: Maximum rank count.
[-N ranks_per_node]: Specify the number of ranks per node and indicate that the order should try to pack nearest neighbors within runs of size ranks_per_node.
[-n ranks_per_line]: Specify an alternate number of ranks per line.
[-q]: Suppresses comment lines in output so that only rank order is produced.
[-s starting_rank]: Specify the rank number offset at which to start (default 0).
[-T]: Show a tuple of grid coordinates.
[i1,i2,…]: Show only the ranks at the specified indices.

NOTES

Note that the grid in this discussion is a virtual topology in the application’s logic, not the physical topology of the machine on which the application executes. It is assumed that the ranks in the generated rank order list will be packed onto machine nodes in the order given.

You must specify either -C or -R for column- or row-major numbering.

EXAMPLES

For example, if the application uses a 2 or 3 dimensional grid, use -C if it assigns MPI rank 1 to position (1,0) or (1,0,0), but use -R if it assigns MPI rank 1 to position (0,1) or (0,0,1). To see the difference, compare the output from the following two examples:

$ grid_order -C -g 4,6

$ grid_order -R -g 4,6

The terms may seem backward if (1,0) is interpreted as an x,y coordinate pair, but are natural if interpreted as array indices in Fortran or C. The usage here follows the definition of row-major numbering for a Cartesian virtual topology as described in the MPI standard.

For an application based on an N by M grid that uses column-major numbering and is to be run on nodes that support sixteen ranks each, the following example produces a list of ranks suitable for use with MPICH_RANK_REORDER, such that 4 x 4 blocks nearest neighbors are placed on each node:

$ grid_order -C -c 4,4 -g N,M

If the same application is to be run on nodes that support 32 ranks, you could use:

$ grid_order -C -c 8,4 -g N,M

$ grid_order -C -c 4,8 -g N,M

Whenever possible, order the -c numbers so that each evenly divides the corresponding -g number.

For an N by M by L grid with row-major numbering and 32 ranks per node, one of the following can be used:

$ grid_order -R -c 2,4,4 -g N,M,L

$ grid_order -R -c 4,2,4 -g N,M,L

$ grid_order -R -c 4,4,2 -g N,M,L

In general, if a grid with dimensions N1,N2, …, Nk is tiled with cells of dimensions n1,n2, …, nk, this program lists the ranks in an order that fills successive cells. If the grid is only partially filled, specify a rank count with -m, and ranks beyond that count are omitted from the list.

The order in the list of ranks within cells, and cells within the grid, can be specified with -o and either -Z (default) or -P. The -o specifies, with a permutation of 0,1,…,k-1, the order of the dimensions along which successive cells are filled. The -Z option lists successive rows of cells in the same order, jumping from the end of one row to the beginning of the next, while the -P (Peano) option lists successive rows in alternating order, beginning at alternating ends, like a farmer plowing a field. The -H (Hilbert) option lists ranks in Hilbert curve order.

It is not required that n1 must evenly divide N1, and so on. If the grid cannot be evenly partitioned into full cells of the specified shape, the ranks in the “odd” regions appear in the list in an order determined by tiling with cells trimmed to fit into those regions.

If values i1,i2, … are specified, only the ranks at those indices in the list are shown.

If the -T option is specified, each line shows a tuple of grid coordinates. Otherwise, each line of output shows all the ranks in one cell. The -n option can be used to specify a different number per line.