pat_python

CrayPat Python Sampling and Tracing (BETA)

Author:: Hewlett Packard Enterprise Development LP.
Copyright:: Copyright 2024 Hewlett Packard Enterprise Development LP.
Manual section:: 1

DESCRIPTION

CrayPat Python experiments are Python applications run via pat_run, where the program and [args-for-program] arguments of pat_run follow the usual invocation of a Python program:

$ pat_run -w python3 app.py ... # Python tracing
$ pat_run python3 app.py ... # Python sampling

Python function names appear in pat_report reports with “python.” prepended to the name.

Python sampling experiments support the same breadth of functionality as other CrayPat sampling experiments performed via pat_run. Python tracing experiments support similar functionality, but are specified differently.

Python Tracing

Python callables may be traced using CrayPat’s Regions API or by specifying modules or trace groups via CrayPat runtime environment variables. CrayPat’s Regions API is the easiest way to trace a select callables or regions. To trace callables, use the API’s PAT_trace decorator:

import pat_api # pat_run ensures this module is found
@pat_api.PAT_trace
def user_function():

This creates an API region before and after “user_function”. To trace custom regions, use PAT_region_push and PAT_region_pop:

import pat_api # pat_run ensures this module is found
pat_api.PAT_region_push("Region name in pat_report")
for i in range(5):
      user_function()
pat_api.PAT_region_pop("Region name in pat_report")

For more detail on CrayPat’s Regions API, see the pat_api(1) man page.

To trace entire modules and programming models, use the PAT_RT_TRACE_PYTHON_GROUPS and PAT_RT_TRACE_PYTHON_MODULES environment variables. PAT_RT_TRACE_PYTHON_GROUPS defines a list of predefined trace groups to trace. PAT_RT_TRACE_PYTHON_MODULES defines a list of Python modules to trace. For more detail, see their descriptions in the intro_craypat(1) man page.

Additional Details

Python experiments on programs that use C extension libraries are supported. Such experiments may intermix Python and C function tracing, and leverage usual C experiment functionality.

Python experiments support programs that use Python virtual environments or Anaconda. If the Python executable is statically compiled (e.g. recent Anaconda packages), PAT_RT_PYTHON_DSO must be set. To check if a Python executable is statically compiled, check if it is linked against a “libpython*” shared library:

ldd $(which python3) | grep libpython

If the executable is not linked against a “libpython*” shared library, it is statically compiled.

Additional Restrictions

The following restrictions apply to Python experiments:

MPI must be available for Perftools to coordinate multiprocess activity. The program, however, may use Mpi4Py or NCCL/RCCL (used by PyTorch) to coordinate multiprocess activity. Use of Python’s multiprocessing is not supported.
Support is only available for select Python versions. To view a list of supported versions, load the perftools-base module and execute:
ls $CRAYPAT_ROOT/lib64/lib_pat_python*
For Anaconda’s Python3.10+ installations, only CrayPat’s Regions API may be used for Python tracing.
Mpi4Py’s experimental GPU support is not supported.
Python experiments use libunwind for callstack acquisition. PAT_RT_CALLSTACK_MODE has no effect.
Traced python functions modify the function reference, so equality comparisons between them and other references may change. For example, if user_function is set to the function sound.effects.echo and user_function is traced, “user_function == sound.effects.echo” will be False.
Code in a Python program’s __main__ file may only be traced via CrayPat’s Regions API.

Python Experiment Example: Pynamic

This example uses Slurm, CCE compiler, perftools, and the cray-python modules to run Pynamic. Pynamic is a benchmark utilizing Mpi4Py, Python, and C shared libraries. More information on Pynamic can be found on LLNL’s GitHub.

Clone Pynamic and cd to the appropriate directory:

$ git clone https://github.com/LLNL/pynamic.git
$ cd pynamic/pynamic-pyMPI-2.6a1

Point the linker to the shared libraries that will be used:
export LD_LIBRARY_PATH=".:$LD_LIBRARY_PATH"

Load perftools and cray-python modules:

module load perftools-base
module load perftools-preload
module load cray-python

In config_pynamic.py, replace any references of “gcc” with “cc”.
In Makefile.mpi4py, replace any references of “mpicc” with “cc” and any references of “python-config” with “python3-config”.
In get-symtab-sizes, replace any references of “addall” with “./addall”.
Make a simple Pynamic executable. See Pynamic’s README for argument details:
python3 config_pynamic.py 10 10 --with-cc=cc -u 2 2 -s 42
Ensure the resulting executable, pynamic-mpi4py, was created:
ls pynamic-mpi4py
Use Slurm’s srun command and pat_run to perform the Python experiment and pat_report upon completion to generate a report:
srun pat_run -r ./pynamic-mpi4py \`date +%s`