gloo

Collective communications library with various primitives for multi-machine training.

| Gloo Documentation | PyTorch Distributed Documentation | Introduction to Gloo Presentation |

NOTE: Gloo is considered to be feature complete and in maintenance-only mode. For new usecases or non-bugfix changes please reach out to the maintainers to discuss.

Gloo is a collective communications library. It comes with a number of collective algorithms useful for machine learning applications. These include a barrier, broadcast, and allreduce.

Transport of data between participating machines is abstracted so that IP can be used at all times, or InifiniBand (or RoCE) when available. In the latter case, if the InfiniBand transport is used, GPUDirect can be used to accelerate cross machine GPU-to-GPU memory transfers.

Where applicable, algorithms have an implementation that works with system memory buffers, and one that works with NVIDIA GPU memory buffers. In the latter case, it is not necessary to copy memory between host and device; this is taken care of by the algorithm implementations.

Requirements

Gloo is built to run on Linux and has no hard dependencies other than libstdc++. That said, it will generally only be useful when used in combination with a few optional dependencies below.

Optional dependencies are:

• CUDA and NCCL -- for CUDA aware algorithms, tests, and benchmark
• Google Test -- to build and run tests
• Hiredis -- for coordinating machine rendezvous through Redis
• MPI -- for coordinating machine rendezvous through MPI

Documentation

Please refer to docs/ for detailed documentation.

Building

You can build Gloo using CMake.

Since it is a library, it is most convenient to vendor it in your own project and include the project root in your own CMake configuration.

Test

Building the tests requires Google Test version 1.8 or higher. On Ubuntu, this version ships with version 17.10 and up. If you run an older version, you'll have to install Google Test yourself, and set the GTEST_ROOT CMake variable.

You can install Google Test using conda with:

conda install -c anaconda gmock gtest

Be carefull that you might need to fish for a package that works with your glibc

To build the tests, run:

mkdir -p build
cd build
cmake ../ -DBUILD_TEST=1 -DGTEST_ROOT=/some/path (if using custom install)
make
ls -l gloo/test/gloo_test*

To test the CUDA algorithms, specify USE_CUDA=ON as well, and the CUDA tests are built at gloo/test/gloo_test_cuda.

Benchmark

First install the dependencies required by the benchmark tool. On Ubuntu, you can do so by running:

sudo apt-get install -y libhiredis-dev

Then build the benchmark, run:

mkdir build
cd build
cmake ../ -DBUILD_BENCHMARK=1
make
ls -l gloo/benchmark/benchmark

Benchmarking

The benchmark tool depends on Redis/Hiredis for rendezvous. The benchmark tool for CUDA algorithms obviously also depends on both CUDA and NCCL.

To run a benchmark:

1. Copy the benchmark tool to all participating machines

2. Start a Redis server on any host (either a client machine or one of the machines participating in the test). Note that Redis Cluster is not supported.

3. Determine some unique ID for the benchmark run (e.g. the uuid tool or some number).

4. On each machine, run (or pass --help for more options):

   bash复制

   ./benchmark \
     --size <number of machines> \
     --rank <index of this machine, starting at 0> \
     --redis-host <Redis host> \
     --redis-port <Redis port> \
     --prefix <unique identifier for this run> \
     --transport tcp \
     --elements <number of elements; -1 for a sweep> \
     --iteration-time 1s \
     allreduce_ring_chunked

Example output (running on 4 machines with a 40GbE network):

…

License

Gloo is BSD-licensed.