nvbandwidth

nvbandwidth

A tool for bandwidth measurements on NVIDIA GPUs.

Measures bandwidth for various memcpy patterns across different links using copy engine or kernel copy methods. nvbandwidth reports current measured bandwidth on your system. Additional system-specific tuning may be required to achieve maximal peak bandwidth.

Requirements

Linux is the primary documented platform. Windows builds are possible but may require additional setup (see Build below).

The following components are required:

• CUDA Toolkit (version 11.x or above). Multinode version requires 12.3 toolkit and 550 driver or above.
• C++17 compiler. GCC 7.x or above on Linux; MSVC 2019 or above on Windows.
• CMake (version 3.20 or above). CMake version 3.24 or newer is encouraged.

On Linux, ensure the path to the nvcc binary (installed via the CUDA Toolkit) is available in $PATH.

The system must have a CUDA-enabled GPU and an NVIDIA display driver compatible with the CUDA Toolkit used to build nvbandwidth. For more information, refer to https://docs.nvidia.com/deploy/cuda-compatibility/

Build

To build the nvbandwidth executable for single-node:

cmake .
make

On Ubuntu/Debian, a utility script is provided that installs the required software components and builds the project:

sudo ./debian_install.sh

On Windows, ensure MSVC and the CUDA Toolkit are on your PATH before running CMake.

Usage:

…

To run all testcases:

./nvbandwidth

To run a specific testcase:

./nvbandwidth -t device_to_device_memcpy_write_ce

Example output:

Running device_to_device_memcpy_write_ce.
memcpy CE GPU(row)  (UUID: GPU-)
GPU 1: NVIDIA  (UUID: GPU-)
...

Across multiple nodes, use mpirun with your hostfile to query all nodes at once. The --map-by ppr:1:node mapping runs one process per node; increase the count to match the number of GPUs per node if you need all UUIDs:

# One process per GPU — shows all GPUs on every node (e.g. 4 GPUs/node, 2 nodes = -n 8)
mpirun --hostfile /etc/nvidia-imex/nodes_config.cfg \
    --map-by ppr:4:node -n 8 \
    nvidia-smi -L

Note that output from different nodes may be interleaved; redirect per-host output with --output-filename if you need clean separation.

--pair affects the following testcase families:

• device_to_device_memcpy_*
• device_to_device_bidirectional_memcpy_*
• device_to_device_latency_sm / device_to_device_latency_tma
• multinode_device_to_device_memcpy_*
• multinode_device_to_device_bidirectional_memcpy_*

Buffer size

A small buffer (e.g. -b 1) is sufficient for quickly verifying connectivity between two GPUs. Use the default 512 MiB buffer (or omit -b entirely) when measuring peak bandwidth, as smaller buffers may underreport achievable throughput.

Single-node pairwise test

Pass both UUIDs directly. The matrix output will show only the row/column entries for those two devices; all other pairs are skipped.

./nvbandwidth -t device_to_device_memcpy_read_sm \
    --pair  

Example output:

Running device_to_device_memcpy_read_sm.
memcpy SM GPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3
 0       N/A       N/A       N/A    

SUM device_to_device_memcpy_read_sm 
COEFFICIENT_OF_VARIATION device_to_device_memcpy_read_sm 

Multinode pairwise test

For multinode tests, run with exactly 2 MPI ranks (one per GPU). Each rank's UUID must be the GPU that rank will use. A hostfile pointing to the two nodes of interest must already be configured for MPI.

mpirun -n 2 ./nvbandwidth -t multinode_device_to_device_memcpy_read_sm \
    --pair  

Example output:

Running multinode_device_to_device_memcpy_read_sm.
memcpy SM GPU(row) -> GPU(column) bandwidth (GB/s)
           0         1
 0       N/A    
 1           N/A

SUM multinode_device_to_device_memcpy_read_sm 
COEFFICIENT_OF_VARIATION multinode_device_to_device_memcpy_read_sm 

Multinode benchmarks

In order to build multinode version of nvbandwidth, execute

cmake -DMULTINODE=1 .
make

Multinode nvbandwidth requires MPI. CMake will find a local installation of MPI to build and link against. Multinode also requires installing, setting up the IMEX service, and creating the IMEX channels. IMEX (NVIDIA Internode Memory Exchange) runs on each compute tray to support GPU memory export and import operations across OS domains in an NVLink multi-node deployment. To start the IMEX service, run the following command:

sudo systemctl start nvidia-imex.service Specify the IP addresses of the cluster nodes in /etc/nvidia-imex/nodes_config.cfg file.

For example, to run multinode bandwidth on a system with 2 nodes and 4 GPUs per node run the command: mpirun --allow-run-as-root --map-by ppr:4:node --bind-to core -np 8 --report-bindings -q -mca btl_tcp_if_include enP5p9s0 --hostfile /etc/nvidia-imex/nodes_config.cfg ./nvbandwidth -p multinode

Note: enP5p9s0 is a site-specific network interface name. Replace it with the appropriate interface on your cluster (e.g. the output of ip link show), or omit -mca btl_tcp_if_include entirely if your MPI environment or job scheduler already handles network binding.

Pair Sampling for Multinode Tests

For large multinode systems, testing all possible GPU pairs can be time-consuming and resource-intensive. nvbandwidth provides a sampling option to reduce test time while maintaining good coverage of the GPU topology.

Sampling Options

• --targetNumPairs -1 (default): Test all possible pairs (N×(N-1) for N GPUs)
• --targetNumPairs : Test exactly `` pairs using intelligent sampling

For example, to run multinode bandwidth on a system with 4 nodes and 8 GPUs per node (32 GPUs total), and select 8 pairs of GPUs run the command: mpirun --allow-run-as-root --map-by ppr:8:node --bind-to core -np 32 --report-bindings -q -mca btl_tcp_if_include enP5p9s0 --hostfile /etc/nvidia-imex/nodes_config.cfg ./nvbandwidth -p multinode --targetNumPairs 8

Note: Replace enP5p9s0 with your cluster's network interface name (see note above).

The command selects random GPU pairs, maximizing coverage of unique GPUs. Bandwidth (BW) for untested pairs will be set to NA.

Sampling Algorithm

When using sampling (targetNumPairs > 0 and less than total pairs), nvbandwidth employs a two-phase approach:

1. GPU Coverage Phase: Ensures each GPU participates in at least one test pair
2. Random Filling Phase: Fills remaining slots with random pairs, prioritizing uncovered GPUs

This approach maximizes GPU topology coverage even with limited test pairs.

Examples

# Test all pairs (default/full coverage) on a 4-node, 8-GPU/node system (32 ranks)
mpirun -n 32 ./nvbandwidth -p multinode --targetNumPairs -1

# Test 20 carefully selected pairs
mpirun -n 32 ./nvbandwidth -p multinode --targetNumPairs 20

# For a 4-node, 8-GPU/node system: 992 total pairs available (32×31)
# Using --targetNumPairs 100 will test ~10% of pairs while covering all GPUs

Note: The --targetNumPairs parameter only affects multinode device-to-device tests. In single-node mode, this parameter is ignored and a warning will be displayed if a positive value is specified.

Local testing

You can test it on a single-node machine (Ampere+ GPU required):

mpirun -n 4 ./nvbandwidth -p multinode

This command will spawn 4 processes, and run all tests with "multinode" prefix.

Running it on a cluster

To run it on a cluster, submit a job to a workload scheduler that has MPI integration. Run one process per GPU.

Running less processes than GPU count is valid, processes will take consecutive GPUs, starting from GPU 0.

Running more processes than GPU count is not valid.

All ranks in the MPI batch must be part of one multinode clique. Run one MPI rank per GPU (typically one rank per GPU per node).

When running under MPI, only MPI rank 0 will output stdout to the console. Stderr, if needed, will be output by all processes.

It is recommended to only run "multinode*" testcases under MPI. While any testcase will succeed, results for non multinode testcases will only come from MPI rank 0.

Test Details

There are two types of copies implemented, Copy Engine (CE) or Streaming Multiprocessor (SM)

CE copies use memcpy APIs. SM copies use kernels.

SM copies will truncate the copy size to fit uniformly on the target device to correctly report the bandwidth. The actual byte size for the copy is:

(threadsPerBlock * deviceSMCount) * floor(copySize / (threadsPerBlock * deviceSMCount))

threadsPerBlock is set to 512.

Latency Measurements

nvbandwidth uses pointer chasing to measure memory latency rather than simple sequential access patterns. This methodology provides more realistic latency measurements by forcing random memory access patterns that prevent prefetching optimizations.

How Pointer Chasing Works

1. Setup: Memory is organized as a linked list where each node contains a pointer to the next node
2. Pattern: The chain follows a strided pattern through memory: Node[i] -> Node[(i + stride) % total_nodes]
3. Execution: The kernel follows this pointer chain for a specified number of accesses
4. Measurement: Total time is divided by number of accesses gives latency per access

Important Notes

• TLB costs are not attributed to latency in these measurements: the same buffer is reused across all accesses, keeping TLB entries warm so that address translation overhead does not appear in the reported latency numbers. TLB lookups still occur in hardware; they are simply not what this microbenchmark is measuring.
• Data cache hits prevented: Random pointer chasing patterns prevent data cache benefits by design

This approach provides latency measurements that benefit from TLB optimization while maintaining random memory access patterns that prevent data cache effects.

Measurement Details

A blocking kernel and CUDA events are used to measure time to perform copies via SM or CE, and bandwidth is calculated from a series of copies.

First, we enqueue a spin kernel that spins on a flag in host memory. The spin kernel spins on the device until all events for measurement have been fully enqueued into the measurement streams. This ensures that the overhead of enqueuing operations is excluded from the measurement of actual transfer over the interconnect. Next, we enqueue a start event, certain count of memcpy iterations, and finally a stop event. Finally, we release the flag to start the measurement.

This process is repeated 3 times by default (controlled by --testSamples), and the median bandwidth for each trial is reported.

Number of repetitions can be overridden using the --testSamples option, and in order to use arithmetic mean instead of median you can specify --useMean option.

Unidirectional Bandwidth Tests

Running host_to_device_memcpy_ce.
memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s)
          0         1         2         3         4         5         6         7
0     26.03     25.94     25.97     26.00     26.19     25.95     26.00     25.97

Unidirectional tests measure the bandwidth between each pair in the output matrix individually. Traffic is not sent simultaneously.

Bidirectional Host Device Bandwidth Tests

Running host_to_device_bidirectional_memcpy_ce.
memcpy CE CPU(row)  GPU(column) bandwidth (GB/s)
          0         1         2         3         4         5         6         7
0     18.56     18.37     19.37     19.59     18.71     18.79     18.46     18.61

The setup for bidirectional host to device bandwidth transfer is shown below:

CE copies
Stream 0 (measured stream) performs writes to the device, while the interfering stream in the opposite direction produces reads. This pattern is reversed for measuring bidirectional device to host bandwidth as shown below.

SM copies
Th