mpich

MPICH

MPICH is a high-performance and widely portable implementation of the Message Passing Interface (MPI) standard from the Argonne National Laboratory. This implementation provides all MPI functions and features required by the standard with comprehensive support for parallel computing applications.

MPICH is widely adopted by multiple vendor partners including Intel, Hewlett Packard Enterprise (HPE), ParTec, Ohio State University (X-ScaleSolutions), and has been deployed on many Top 500 HPC systems.

This README file should contain enough information to get you started with MPICH. More extensive installation and user guides can be found in the doc/installguide/install.pdf and doc/userguide/user.pdf files respectively in the release tarball, or the doc/wiki in the source code.

Table of Contents

1. Getting Started
2. CH4 Netmods
3. GPU Support
4. Threading Supports
5. Testing the MPICH Installation
6. Reporting Installation or Usage Problems
7. Developer Builds
8. Compiler Flags
9. Multiple Fortran Computer Support
10. ABI Compatibility
11. Citing MPICH

1. Getting Started

Note: this guide assumes you are building MPICH from a release tarball. If you are starting from a git checkout, you will need a few additional steps. Please refer to the wiki page.

The following instructions take you through a sequence of steps to get the default configuration (CH4 device with OFI network module, Hydra process management) of MPICH up and running.

Prerequisites

You will need the following prerequisites:

• REQUIRED: The MPICH source tarball

• REQUIRED: Perl

• REQUIRED: A C compiler (C99 support is required. See Shifting Toward C99, C11 is preferred for C11 Atomics)

• OPTIONAL: A C++ compiler, if C++ applications are to be used (g++, etc.). If you do not require support for C++ applications, you can disable this support using the configure option --disable-cxx (configuring MPICH is described in step 1(d) below).

• OPTIONAL: A Fortran compiler, if Fortran applications are to be used (gfortran, ifort, etc.). If you do not require support for Fortran applications, you can disable this support using --disable-fortran (configuring MPICH is described in step 1(d) below).

• OPTIONAL: Python 3. Python 3 is needed to generate Fortran bindings.

Note: For users of csh and tcsh shells, please refer to Using csh and tcsh Shells for shell-specific examples.

Installation Steps

(a) Unpack the tar file and go to the top level directory:

tar xzf mpich-.tar.gz
cd mpich-

(b) Choose an installation directory

Choose an installation directory, say /home//mpich-install, which is assumed to non-existent or empty. It will be most convenient if this directory is shared by all of the machines where you intend to run processes. If not, you will have to duplicate it on the other machines after installation.

(c) Configure MPICH

Configure MPICH specifying the installation directory and device:

./configure --prefix=/home//mpich-install 2>&1 | tee c.txt

The configure will try to determine the best device (the internal network modules) based on system environment. You may also supply a device configuration. E.g.

./configure --prefix=... --with-device=ch4:ofi |...

or:

./configure --prefix=... --with-device=ch4:ucx |...

Refer to section below -- CH4 Netmods -- for more details.

If a failure occurs, the configure command will display the error. Most errors are straight-forward to follow. For example, if the configure command fails with:

"No Fortran compiler found. If you don't need to build any Fortran programs, you can disable Fortran support using --disable-fortran. If you do want to build Fortran programs, you need to install a Fortran compiler such as gfortran or ifort before you can proceed."

... it means that you don't have a Fortran compiler :-). You will need to either install one, or disable Fortran support in MPICH (with --disable-fortran option).

If you are unable to understand what went wrong, please go to Section (6) below, for reporting the issue to the MPICH developers and other users.

(d) Build MPICH:

make 2>&1 | tee m.txt

This step should succeed if there were no problems with the preceding step. Check file m.txt. If there were problems, do a make clean and then run make again with V=1 to get the verbose output.

make V=1 2>&1 | tee m.txt

(e) Install the MPICH commands:

make install 2>&1 | tee mi.txt

This step collects all required executables and scripts in the bin subdirectory of the directory specified by the prefix argument to configure.

(f) Add the bin subdirectory to your path

Add the bin subdirectory of the installation directory to your path in your startup script (.bashrc for bash, etc.):

PATH=/home//mpich-install/bin:$PATH ; export PATH

Check that everything is in order at this point by doing:

which mpicc
which mpiexec

These commands should display the path to your bin subdirectory of your install directory.

IMPORTANT NOTE: The install directory has to be visible at exactly the same path on all machines you want to run your applications on. This is typically achieved by installing MPICH on a shared NFS file-system. If you do not have a shared NFS directory, you will need to manually copy the install directory to all machines at exactly the same location.

(g) Run an MPI job

MPICH uses a process manager for starting MPI applications. The process manager provides the "mpiexec" executable, together with other utility executables. MPICH comes packaged with multiple process managers; the default is called Hydra.

Now we will run an MPI job, using the mpiexec command as specified in the MPI standard. There are some examples in the install directory, which you have already put in your path, as well as in the examples directory of your MPICH installation. One of them is the classic CPI example, which computes the value of pi by numerical integration in parallel.

To run the CPI example with 'n' processes on your local machine, you can use:

mpiexec -n  ./examples/cpi

Test that you can run an 'n' process CPI job on multiple nodes:

mpiexec -f machinefile -n  ./examples/cpi

The 'machinefile' is of the form:

host1
host2:2
host3:4   # Random comments
host4:1

'host1', 'host2', 'host3' and 'host4' are the hostnames of the machines you want to run the job on. The ':2', ':4', ':1' segments depict the number of processes you want to run on each node. If nothing is specified, ':1' is assumed.

More details on interacting with Hydra can be found at Using the Hydra Process Manager

If you have completed all of the above steps, you have successfully installed MPICH and run an MPI example.

2. CH4 Netmods

MPICH uses the CH4 device by default, which supports different network modules (netmods) for communication. This section covers the available CH4 netmods and their configuration options.

Note: For information about alternate devices and channels (including legacy CH3), please refer to Alternate Channels and Devices.

OFI Netmod

The ofi netmod provides support for the OFI network programming interface. To enable, configure with the following option:

--with-device=ch4:ofi[:provider]

If the OFI include files and libraries are not in the normal search paths, you can specify them with the following options:

--with-libfabric-include= and --with-libfabric-lib=

... or the if lib/ and include/ are in the same directory, you can use the following option:

--with-libfabric=

If specifying the provider, the MPICH library will be optimized specifically for the requested provider by removing runtime branches to determine provider capabilities. Note that using this feature with a version of the libfabric library older than that recommended with this version of MPICH is unsupported and may result in unexpected behavior. This is also true when using the environment variable FI_PROVIDER.

Please refer to the MPICH documentation for the currently recommended version of libfabric.

UCX Netmod

The ucx netmod provides support for the Unified Communication X library. It can be built with the following configure option:

--with-device=ch4:ucx

If the UCX include files and libraries are not in the normal search paths, you can specify them with the following options:

--with-ucx-include= and --with-ucx-lib=

... or the if lib/ and include/ are in the same directory, you can use the following option:

--with-ucx=

By default, the UCX library throws warnings when the system does not enable certain features that might hurt performance. These are important warnings that might cause performance degradation on your system. But you might need root privileges to fix some of them. If you would like to disable such warnings, you can set the UCX log level to "error" instead of the default "warn" by using:

UCX_LOG_LEVEL=error
export UCX_LOG_LEVEL

Capability Sets for CH4 OFI Netmod

Capability sets simplify CH4 OFI netmod configuration by providing optimized presets for specific OFI providers.

Compile-time Configuration (Recommended for Performance)

For best performance, specify the OFI provider during configuration:

--with-device=ch4:ofi:sockets

This optimizes MPICH for the sockets provider and sets compile-time constants that cannot be changed at runtime.

Runtime Configuration (More Flexible)

For flexibility, configure without specifying the provider:

--with-device=ch4:ofi

Then set the provider and options at runtime:

export FI_PROVIDER=sockets
export MPIR_CVAR_CH4_OFI_ENABLE_ATOMICS=0  # Example: disable atomics

Configuration Variables

Common capability set variables:

• MPIR_CVAR_CH4_OFI_ENABLE_ATOMICS - Enable/disable provider atomics
• MPIR_CVAR_CH4_OFI_ENABLE_TAGGED - Enable/disable tagged messaging
• MPIR_CVAR_CH4_OFI_ENABLE_RMA - Enable/disable RMA operations

See README.envvar for the complete list of configuration variables.

3. GPU Support

MPICH automatically detects and enables GPU support for CUDA, Intel Level Zero (ZE), and HIP runtimes during configuration.

Automatic Detection

GPU support is automatically enabled if any supported GPU runtime is detected during the configure process.

Manual Configuration

To specify custom GPU runtime locations:

--with-cuda=     # For NVIDIA CUDA
--with-ze=       # For Intel Level Zero
--with-hip=      # For AMD HIP

For separate lib/ and include/ directories:

--with-cuda-include= --with-cuda-lib=

Disabling GPU Support

To explicitly disable GPU support:

--without-cuda  # Disable CUDA
--without-ze    # Disable Intel Level Zero
--without-hip   # Disable AMD HIP

Runtime Control

GPU support can be disabled at runtime to avoid initialization overhead for non-GPU applications:

export MPIR_CVAR_ENABLE_GPU=0

4. Threading Supports

MPICH default enables the support for threading with all four MPI threading levels (SINGLE, FUNNELED, SERIALIZED, MULTIPLE).

The supported thread level are configured by opti