Environment Modules for HPE CPE Supercomputing Systems

Overview

This documentation is targeted at CPE customers using a module environment. CPE environment module programs are used to manage a user’s environment. A CPE system can be setup with either Lmod or TCL Modules.

The following information assumes the reader has some familiarity with modules and CPE environments. (If you are a beginner, studying the open source Lmod and Environment Modules documentation is highly recommended.)

The following covers the topics:

  • How environment modules are designed to function within the CPE environment

  • What key differences exist between Lmod and TCL Modules

  • How to setup a modules environments for Lmod and TCL Modules

  • How to swap programing environments

  • How to setup custom modules hierarchy using (Lmod only)

This document does not cover the basic knowledge of what environment modules are or how to run specific CPE programs using environment modules.

Terms

The below terms shall be used throughout this document as defined below. These definitions are necessary, given the ubiquitous nature of the term “module” and the original module program called “Environment Modules.”

module-file

A file that contains information to configure a user’s environment for the specific application they represent (i.e. the module-file representing Cray Mpich is “cray-mpich”). The information contained in the module-file may include the setting of environment variables (i.e. MANPATH, PATH, etc).

environment module program

Is a tool that uses files referred to as module-files (also called “environment modules” or simply “modules”) to dynamically modify a user environment. The module program needs to be installed and initialized before an environment can be modified by module-files. There are two environment module programs a CPE system may use: Lmod or TCL Modules.

Lmod

An environment module program that uses environment modules written in Lua. Lmod can use a directory hierarchy to manage module dependancies.

TCL Modules

The original environment module program called “Environment Modules” with module-files written in TCL. CPE’s TCL Modules has been built from the original open source C code base of “Environment Modules” with Cray features built into it.

environment modules

This term my refer to either Lmod modules, TCL Modules, or both TCL Module and Lmod module-files collectively.

module

A Lmod or TCL Module module-file

Other Module Resources

CPE Documentation

  • CPE Installation Guide

  • CPE User Guide

  • CPE Release Announcements

External Resources

Module Environments Design Overview

How environment modules are designed to function within the CPE environment

System setup decisions:

A CPE system can be setup with either Lmod or TCL Modules (not both).

  • Use the configuration files to set the default modules desired upon user login.

  • Set the default versions to be used on a system.

  • (Lmod only) Join customer Lmod hierarchies with the CPE Lmod hierarchy to make both directory trees act as one hierarchy on a system for all users.

    Note: For more details see System Module Environment Setup

CPE package input requirements for environment module programs:

  • Each CPE package is required to support Lmod and TCL Modules. This is accomplished by the product package containing a Lua and TCL module-file. Each module-file contains environment settings information specific to the CPE product it represents. (i.e. Adding a program path to “PATH” or setting a new environment variable to act as a program flag.)

  • Each CPE package must install both TCL Modules and Lmod module-files in specific module path locations. Lmod module-file location is more complex since Lmod’s hierarchy features require a “nested” directory tree to function.

  • Each CPE package is also required to support “Set Default” which manages the different versions of CPE products and their module-files.

    Note: The above settings are configured by a system administrator. For system specific details, please see HPE CPE Supercomputing Install Guide for your system.

CrayPE Driver functionality inside module environments:

1 This driver reads in the current state of environment variables set by the current module-files loaded. For example, the PE_ENV environment variable is set by the PrgEnv-<environment> modules and conveys the current programming environment set (Cray, GNU, etc.).

2 The CrayPE Driver then maps the current programing environment variables to the currently loaded targeting modules’ (i.e. module beginning with “craype-accel-”, “craype-network-”, and “craype-<cpu_name>-”) and sets defines for pkgconfig. Part of this step involves the setting the volatile values found in some CPE module-files. Note: TCL Modules are more reliant on the CrayPE Driver than Lmod; however, both environment module programs use the CrayPE Driver to populate volatile variables within the module-files at present.)

3 Lastly, the CrayPE Driver outputs the collected data into a coherent command-line for the current programming environment (i.e. Cray, GNU, etc.). To invoke the CrayPE Driver use the symlinks cc, CC, or ftn. The command-line has the below structure:

  • The first command-line option to the filename is what the CrayPE Driver adds

  • The filename is from the user’s command-line

  • Everything after the filename is from pkgconfig (sdriver_pkgconfig.c). pkgconfig sets defaults and links things together.

To see an example of the command-line generated by the CrayPE Driver run:

cc CC ftn <filename> -craype_verbose

Lmod vs TCL Modules

Key differences between Lmod and TCL Modules

TCL Modules are the original environment module program. Lmod is based on Environment Modules and it provides many of the same features to dynamically modify a user environment. This section concentrates on the key differences between Lmod and TCL Modules.

Use different languages: - TCL modules is written in C with module-files written in TCL - Lmod is implemented in Lua

Hierarchical module environments:

  • TCL Modules require the user to have some sense of which modules pair with one another. It shows the user all the modules available, regardless of if they would work if loaded in the present environment state. CPE uses the CrayPE Driver to assist in auto correcting modules to align behind the scenes.

  • Lmod uses a directory hierarchy. As core modules are loaded, they have the ability to add new paths to the MODULEPATH. This behavior allows Lmod module-files to manage their dependancies and ensure all module-files a user can load in the present state are compatible.

Conflicts:

  • TCL Modules use module conflicts to convey that two or more modules cannot be loaded together. The conflict must be coded into each module-file that has a conflict.

  • Lmod can use standard module conflicts like TCL Modules and the Lmod command family to enable circular dependancies. For example: family(“craype_accel”) can be added to every GPU target module. This will ensure only one GPU target module will be loaded at a time, even if the module-file comes from a customer Lmod hierarchy.

Autoswapping:

  • TCL Modules must be manually swapped out.

  • Lmod modules can be swapped out by loading the module desired. This means the currently conflicting modules that are currently loaded along with all of their dependancies will be auto swapped out in favor of the loading module.

Note: Please note one level Lmod auto swapped modules are easily handled. For more complex multi-level auto swapped modules (such as swapping full programming environments) it is better to use the command swap. This is due to the load order difference between a module load auto-swap and a module swap with multilevel dependancies being changed out.

Components

Setup System Modules

Lmod Custom Dynamic Hierarchy

Using Modules