intro_ffio

Date:: 10-20-2011

NAME

intro_ffio - Describes performance options available with the FFIO layers

IMPLEMENTATION

Cray Linux Environment (CLE)

DESCRIPTION

The Flexible File I/O (FFIO) system lets the user specify a comma-delimited list of layers through which I/O data is to be passed. This is done by providing a value for the spec argument to the -F option on the assign(1) command. This specifies a class of processing to be done on the data.

The following C routines are used with FFIO:

fffcntl(3c): Performs functions on files opened using FFIO.
fflistio(3c): Initiates a list of I/O requests using FFIO.
ffiolock(3c): Locks and unlocks function calls.
ffopen(3c): Opens or closes a file using FFIO.
ffread(3c): Reads from a file using FFIO.
ffseek(3c): Repositions an FFIO file.
ffwrite(3c): Writes to a file using FFIO.

Specifying FFIO Layers

The spec argument of the -F option of the assign(1) command comprises a list of layers or filters that are used to manipulate the data file as it is being read or written. The available layers include performance options (such as memory-resident files) and the capability to read and write files in a variety of different vendors’ blocking formats. Each layer spec is of the general form:

class[.*type*[.*subtype*][.num1][.num2][.num3]]

Many of the layers also allow you to specify the numeric parameters with a keyword. This requires that your application be linked with CrayLibs 3.0 release or later releases. The format for these specifications is given in the description of each layer where the specification is available.

The spec argument can have the following values:

bufa: Asynchronous buffering layer.

The bufa layer provides asynchronous buffering. It allows efficient sequential-access I/O.
The num1 field represents the size in 4096-byte blocks of each buffer.
The num2 field selects the number of buffers to be used.
You can specify the numeric parameters with this alternate keyword syntax:
bufa[.bufsize=num1][.num_buffers=num2]

cache: Cached file. The cache layer allows efficient random-access I/O, even when file access is clustered in several regions of a file.

During reads and writes to the layer, cache buffers frequently must be preempted. The buffer chosen for preemption is always the least recently accessed buffer at the time of preemption.
The option available is .mem, which specifies that buffers reside in memory. Memory-resident buffers are the default.
The numeric fields are as follows:
- num1 is the size in 4096-byte blocks of each cache page.
- num2 selects the number of cache pages to be used.
- num3 is the size in 4096-byte blocks at which the cache layer attempts to bypass cache layer buffering. If a user’s I/O request is larger than num3, the request might not be copied to a cache page.
You can specify the numeric parameters with this alternate keyword syntax:
cache[.*type*][.page_size=num1][.num_pages=num2][.bypass_size=num3]

cachea: Asynchronously cached file.

This type of processing usually performs well whenever the cache layer might be used. In addition, any sequential forward and sequential backward access through the file is detected. When sequential access patterns are detected while reading, asynchronous read-ahead is performed provided that the numbers of pages to read ahead has been specified. When writing, selective asynchronous write-behind is performed.
The value for type is .mem, which specifies that cache pages reside in memory.
The numeric fields are as follows:
- num1 specifies the size in 4096-byte blocks of each cache page buffer. The default is 512.
- num2 selects the number of cache pages to be used. The default is 8.
- num3 selects the number of pages to read ahead asynchronously. The default is either num2-1 or 8, whichever is smaller.
You can specify the numeric parameters with this alternate keyword syntax:
cachea[.*type*][.page_size=num1][.num_pages=num2][.max_lead=num3]

cos: COS blocking.

If specified, type must be one of the following:

sync
Uses a single buffer in the blocking and deblocking process. I/O is done strictly synchronously.

async
Divides the buffer into two parts and uses asynchronous I/O to transfer the blocked data between the buffer(s) and the logical device. When reading, asynchronous read-ahead is performed, and when writing, asynchronous write-behind is performed. To effectively use async, the buffer size should be at least twice the record length.

auto
Default (if type is not specified). Chooses either synchronous or asynchronous behavior depending on the buffer size. If the buffer size is less than 64 blocks, synchronous behavior is selected. If it is greater than or equal to 64 blocks, asynchronous behavior is selected.
For num1, enter the desired buffer size in 4096-byte blocks (for example,-F cos:42 requests COS blocking and a 42-block buffer). The num1 value also determines the record size for underlying layers which perform record blocking. The underlying record size is num1 blocks if in synchronous mode and num1/2 or num1/2+1 blocks if in asynchronous mode.
You can specify the numeric parameters with this alternate keyword syntax:
cos[.*type*][.bufsize=num1]
Besides the .bufsize argument shown above the environment variable FF_COS_DEFAULT_BUFFER_SIZE also controls buffer sizes in the COS blocked layer. If the environment variable is set to the value of FIXED, the buffer size for the COS blocked layer is set to a static default value of 48 sectors; otherwise, a variable buffer size is used.

event: I/O layer monitoring.

The event layer monitors I/O occurring between two layers on a per-file basis and generates information in a text log file. You can specify what type of report is generated. The event layer is enabled by default; you do not have to relink your program to study I/O performance. To generate information, rerun the program with the event layer specified on the assign command line at the desired position(s) in the FFIO call chain.
Information is written to stderr by default. The FFIO_EVENT_LOGFILE environment variable can be used to select an alternate file to which the reports are written. The default action is to overwrite the existing file. To append information to an existing file, specify a plus sign (+) immediately before the file name.
The event layer counts all I/O (read, write, etc.) and all I/O-related (open, close, fcntl, etc.) requests. These counts represent the number of calls made by the parent layer above the event layer to the child layer below the event layer. (Note that ‘above’ and ‘below’ are somewhat arbitrary designations, with the ‘higher’ layers being closer to the program or application and the ‘lower’ layers being closer to the operating system.) Both the numbers and types of requests can change as you move down the FFIO chain, as FFIO layers will consolidate multiple I/O requests into fewer requests and convert requests from one type to another (e.g., synchronous to asynchronous).
Enter one of the following for type:
nostat
No statistical information is reported.

brief
Generates a report on the amount of data transferred through the event layer.

summary
Generates three reports:
- A report on the amount of data transferred (same as the brief option, above)
- A report on file information, including the file size
- A list of all the I/O and I/O-related requests that passed through the event layer
The default type is summary.

f77: Fortran record blocking. This is the common blocking format used by most Fortran compilers on systems.

Enter one of the following options for type:

nonvax
Default. Control words in a format common to machines such as the MC68000.

vax
VAX format (byte-swapped) control words.
The specification of vax or nonvax is not relevant to data conversion.
num1
Maximum record size in bytes. The default is 2 MB. The maximum value that can be specified is 4 MB.

num2
Buffer size in bytes. The default is 65 KB.
You can specify the numeric parameters with this alternate keyword syntax:
f77[.*type*][.recsize=num1][.bufsize=num2]

fd: Connect to a specific file descriptor.

Field num1 is the decimal value of the file descriptor. Classes named stdin, stdout, and stderr exist as alternate names for fd:0, fd:1, and fd:2.
You can specify the numeric parameters with this alternate keyword syntax:
fd[.file_descriptor=num1]

global: (Deferred implementation) File global to all processes.

This is a caching layer which distributes the cache across multiple SHMEM or MPI processes. Open and close operations are collective (require participation by all processes which access the file). All other operations are independently performed by one or more processes. The library allows multiple processes to concurrently access the file while maintaining coherency of buffered data.
Specify one of the following options for type:

privpos
Default. The file position is private to a process. All processes may seek to independent locations in the file.

globpos
The file position is global to all processes. A seek or I/O operation by any process will affect the position for all processes.
The numeric fields are as follows:

num1
The size in 4096-byte blocks of each cache page.

num2
Selects the number of cache pages to be used on each process. If there are n processes, then n* num2 cache pages are used.
- num2 buffer pages are allocated on every process which shares access to a global file. File pages are direct-mapped onto processes such that page n of the file will always be cached on process (nmod NPES), where NPES is the total number of processes sharing access to the global file. Once the process is identified where caching of the file page will occur, a least-recently-used method is used to assign the file page to a cache page within the caching process.
You can specify the numeric parameters with this alternate keyword syntax:
global[.*type*][.page_size=num1][.num_pages=num2]

ibm

Record blocking for common record types on IBM operating systems. Specify one of the following record formats for type:

f: Fixed-length record format. For num1, enter the logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes; if specified, num2 must be equal to num1.
fb: Fixed-length, blocked record format. For num1, enter the logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes; num2 must be an exact multiple of num1.
u: Undefined record format. For num1, enter the maximum record size, in 8-bit bytes.
v: Variable length record format. For num1, enter the maximum logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes.
vb: Variable length blocked record format. For num1, enter the maximum logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes.
vbs: Variable length blocked, spanned record format. For num1, enter the maximum logical record size in 8-bit bytes. For num2, enter the maximum physical block size in 8-bit bytes.

No subtype is accepted for the ibm class record formats. num1 does not need to be smaller than num2.
You can specify the numeric parameters with this alternate keyword syntax:
ibm[.*type*][.recsize=num1][.mbs=num2]

mr: Memory-resident files.

The mr layer lets you declare that all or part of a file will reside in memory. This can improve performance for relatively small files that are heavily accessed or for larger files where the first part of the file is heavily accessed (for example, a file which contains a frequently updated directory at the beginning).
Enter one of the following values for type:

save
Default. The file is loaded into memory when the file is opened and written back out to disk when the file is closed.

scr
Scratch file. The file is not read into memory when opened and is not written to disk when closed.
Enter one of the following values for subtype:

ovfl
Default. Data which does not fit (overflows) the maximum specified memory allocation is written to the next lower layer (typically a disk file). An informative message is written to stderr on the first overflow.

ovflnomsg
As above, but no informative message is issued when the data overflows the memory-resident buffer.

novfl
If data does not fit in memory then subsequent write(1) operations fail.
The mr layer accepts numeric fields. Generally, the numeric field specification represents best-efforts values. They are used for tuning purposes and usually do not cause errors if they are not satisfied precisely as specified. The fields are as follows:

num1
Initial size of the memory allocation, specified in 4096-byte blocks. The default is 0.

num2
Maximum size of the memory allocation, specified in 4096-byte blocks. The default is 256 blocks (1 MB) or num1, whichever is larger.

num3
Increment size of the memory allocation, specified in 4096-byte blocks. This value is used when allocating additional memory space. The default is 256 blocks (1 MB) or (num2 - num1), whichever is smaller.
When using memory-resident files, you must ensure that the size of the memory-resident portion of the file is limited to reasonable values. Unrestrained and unmanaged growth of such a file portion can cause heap fragmentation, exhaustion of all available memory, and program abort. If this growth has consumed all available memory, the program may not abort gracefully, making such a condition difficult to diagnose.
Increment sizes which are too small can also contribute to heap fragmentation.
You can specify the numeric parameters with this alternate keyword syntax:
mr[.*type*[.*subtype*]][.start_size=num1] [.max_size=num2] [.inc_size=num3]

null: Syntactic no-op.

No optional fields are accepted. null may be specified where syntax demands a value but no function is desired. This does not perform the same function as /dev/null.

site

Site layer. This option lets a site add custom I/O handlers for specific needs at load time. See the Cray Fortran Reference Manual for details.

stdin, stdout, stderr

Connects to specific file descriptors 0, 1, and 2, respectively. (See the fd class.)

system

Generic system I/O layer. Select er90 or syscall as appropriate.

syscall

System I/O call. Each I/O operation results in a corresponding system call.

text

Special character-terminated records. Enter one of the following for type:

nl: Newline-separated records.
eof: Newline-separated records and the special character sequence ~e on a line by itself delimiting EOF.

This class accepts num specifications. If specified, num1 represents the decimal value of the ASCII character to use to delimit records; the default varies depending on the type. For num2, enter the requested working buffer size in bytes.
You can specify the numeric parameters with this alternate keyword syntax:
text[.*type*][.newline=num1][.bufsize=num2]

user

User layer. This option allows a user to add custom I/O handlers for specific needs at load time. See the Cray Fortran Reference Manual for details.

vms

Provides record blocking for common record types on VMS/VMS operating systems. For type, enter one of the following record formats:

f: Fixed-length records
v: Variable length records
s: Segmented variable records

Each type accepts the following subtypes to specify the blocking format within the record type:

bb
Binary blocked format. This subtype should be used with files that are to be fetched or disposed with a BB or TB format. This subtype requires an enclosing blocking; for example, vms.s.bb,bbx or vms.s.bb,cos

tr
Transparent format. This subtype should be used with files that are transferred between the VAX/VMS system by using fetch or dispose
- commands and the TR format. Any other method that precisely transfers the disk image, including all VMS control words, will also work. Note that ftp does not correctly transfer nontext, variable-length record files.
This class accepts num1 and num2 fields; they have a similar meaning to ibm class. For type s, num1 is ignored. For type f, num2 need not be a multiple of num1.
You can specify the numeric parameters with this alternate keyword syntax:
vms[.*type*][.subtype][.recsize=num1][.mbs=num2]

NOTES

The Fortran I/O library checks for conflicting attributes when file name and unit attributes are both present during OPEN processing for a Fortran unit. The existence of an assign attribute for both the file and the unit results in an error condition.

EXAMPLES

The following are example FFIO specifications.

Example 1:

The following example specifies Fortran blocking with a working buffer of 128,000 bytes to allow efficient creation of logical records up to 127,992 bytes:

f77::128000

Example 2:

The following example specifies IBM VBS records with a block size of 16,384 bytes:

ibm.vbs::16384

Example 3:

The following example specifies a memory-resident layer with an initial memory size of 100 4096-byte blocks:

mr:100

Example 4:

The following example specifies VMS V records with a maximum record size of 1,000 bytes:

vms.v.tr:1000

Example 5:

The following examples illustrates using the assign command to access big-endian unformatted files on a little-endian system.

assign -N swap_endian foreignfile

This assumes that the file is a normal f77 unformatted file with 32-bit record control images with a byte count. The library routines will swap both the control images and the data when reading or writing the file.

If all unformatted sequential files are the opposite endianness, use the following command:

assign -N swap_endian g:su

See the Cray Fortran Reference Manual for more detailed information about FFIO specifications.