gdrcopy

GDRCopy

A low-latency GPU memory copy library based on NVIDIA GPUDirect RDMA technology.

Introduction

While GPUDirect RDMA is meant for direct access to GPU memory from third-party devices, it is possible to use these same APIs to create perfectly valid CPU mappings of the GPU memory.

The advantage of a CPU driven copy is the very small overhead involved. That might be useful when low latencies are required.

What is inside

GDRCopy offers the infrastructure to create user-space mappings of GPU memory, which can then be manipulated as if it was plain host memory (caveats apply here).

A simple by-product of it is a copy library with the following characteristics:

• very low overhead, as it is driven by the CPU. As a reference, currently a cudaMemcpy can incur in a 6-7us overhead.

• An initial memory pinning phase is required, which is potentially expensive, 10us-1ms depending on the buffer size.

• Fast H-D, because of write-combining. H-D bandwidth is 6-8GB/s on Ivy Bridge Xeon but it is subject to NUMA effects.

• Slow D-H, because the GPU BAR, which backs the mappings, can't be prefetched and so burst reads transactions are not generated through PCIE

The library comes with a few tests like:

• gdrcopy_sanity, which contains unit tests for the library and the driver.
• gdrcopy_copybw, a minimal application which calculates the R/W bandwidth for a specific buffer size.
• gdrcopy_copylat, a benchmark application which calculates the R/W copy latency for a range of buffer sizes.
• gdrcopy_apiperf, an application for benchmarking the latency of each GDRCopy API call.
• gdrcopy_pplat, a benchmark application which calculates the round-trip ping-pong latency between GPU and CPU.

Requirements

GPUDirect RDMA requires NVIDIA Data Center GPU or NVIDIA RTX GPU (formerly Tesla and Quadro) based on Kepler or newer generations, see GPUDirect RDMA. For more general information, please refer to the official GPUDirect RDMA design document.

The device driver requires GPU display driver >= 418.40 on ppc64le and >= 331.14 on other platforms. The library and tests require CUDA >= 6.0.

DKMS is a prerequisite for installing GDRCopy kernel module package. On RHEL or SLE, however, users have an option to build kmod and install it instead of the DKMS package. See Build and installation section for more details.

# On RHEL
# dkms can be installed from epel-release. See https://fedoraproject.org/wiki/EPEL.
$ sudo yum install dkms

# On Debian - No additional dependency

# On SLE / Leap
# On SLE dkms can be installed from PackageHub.
$ sudo zypper install dkms rpmbuild

CUDA and GPU display driver must be installed before building and/or installing GDRCopy. The installation instructions can be found in https://developer.nvidia.com/cuda-downloads.

GPU display driver header files are also required. They are installed as a part of the driver (or CUDA) installation with runfile. If you install the driver via package management, we suggest

• On RHEL, sudo dnf module install nvidia-driver:latest-dkms.
• On Debian, sudo apt install nvidia-dkms-<your-nvidia-driver-version>.
• On SLE, sudo zypper install nvidia-gfx<your-nvidia-driver-version>-kmp.

The supported architectures are Linux x86_64, ppc64le, and arm64. The supported platforms are RHEL8, RHEL9, Ubuntu20_04, Ubuntu22_04, SLE-15 (any SP) and Leap 15.x.

Root privileges are necessary to load/install the kernel-mode device driver.

DMA-BUF mmap backend

GDRCopy can export GPU memory as a Linux dma-buf via the CUDA driver and map it into user space with a plain mmap() on the dma-buf file descriptor. This backend does not require the GDRCopy kernel module (gdrdrv) and is intended for environments where gdrdrv is not installed or loaded.

Requirements

• CUDA driver 13.3 or newer

Backend selection

On gdr_open(), GDRCopy tries gdrdrv first. If gdrdrv is not installed or fails to open, it falls back to the dma-buf mmap backend (provided the driver supports). To force the dma-buf backend even when gdrdrv is available, set environment variable GDRCOPY_USE_DMABUF_MMAP=1 before calling gdr_open(). To check at runtime which backend is active:

int using_dmabuf;
gdr_get_attribute(g, GDR_ATTR_USING_DMA_BUF_MMAP, &using_dmabuf);
// using_dmabuf != 0  -> dma-buf mmap backend is in use

Mapping type

CPU cacheability is decided by the CUDA driver at pin time and cannot be changed afterwards. The mapping type depends on the pin flag and if the platform is coherent.

The dmabuf backend does not support user-requested mapping types: the type the pin produces is the only type gdr_map_v2 will accept. Passing an explicit cache flag (GDR_MAP_FLAG_REQ_CACHE_MAPPING, GDR_MAP_FLAG_REQ_WC_MAPPING, …) that asks for anything other than the default returns EINVAL.

Behavior differences vs. the gdrdrv backend

• Persistent mappings. All dma-buf mappings are persistent; GDR_ATTR_USE_PERSISTENT_MAPPING always returns 1.
• One fd per pinned buffer. Each gdr_pin_buffer consumes one dma-buf file descriptor until gdr_unpin_buffer. Applications that pin many buffers should account for the process FD limit.
• No timing fields. gdr_get_info_v2 returns tm_cycles = 0 and cycles_per_ms = 0.
• No invalidation callback. gdr_get_callback_flag always returns 0.
• GDR API compatibility — Standard GDRCopy APIs remain unchanged

Build and installation

We provide three ways for building and installing GDRCopy.

rpm package

# For RHEL:
$ sudo yum groupinstall 'Development Tools'
$ sudo yum install dkms rpm-build make

# For SLE:
$ sudo zypper in dkms rpmbuild

$ cd packages
$ CUDA=<cuda-install-top-dir> ./build-rpm-packages.sh
$ sudo rpm -Uvh gdrcopy-kmod-<version>dkms.noarch.<platform>.rpm
$ sudo rpm -Uvh gdrcopy-<version>.<arch>.<platform>.rpm
$ sudo rpm -Uvh gdrcopy-devel-<version>.noarch.<platform>.rpm

DKMS package is the default kernel module package that build-rpm-packages.sh generates. To create kmod package, -m option must be passed to the script. Unlike the DKMS package, the kmod package contains a prebuilt GDRCopy kernel module which is specific to the NVIDIA driver version and the Linux kernel version used to build it.

deb package

$ sudo apt install build-essential devscripts debhelper fakeroot pkg-config dkms
$ cd packages
$ CUDA=<cuda-install-top-dir> ./build-deb-packages.sh
$ sudo dpkg -i gdrdrv-dkms_<version>_<arch>.<platform>.deb
$ sudo dpkg -i libgdrapi_<version>_<arch>.<platform>.deb
$ sudo dpkg -i gdrcopy-tests_<version>_<arch>.<platform>.deb
$ sudo dpkg -i gdrcopy_<version>_<arch>.<platform>.deb

from source

$ make prefix=<install-to-this-location> CUDA=<cuda-install-top-dir> all install
$ sudo ./insmod.sh

Notes

Compiling the gdrdrv driver requires the NVIDIA driver source code, which is typically installed at /usr/src/nvidia-<version>. Our make file automatically detects and picks that source code. In case there are multiple versions installed, it is possible to pass the correct path by defining the NVIDIA_SRC_DIR variable, e.g. export NVIDIA_SRC_DIR=/usr/src/nvidia-520.61.05/nvidia before building the gdrdrv module.

There are two major flavors of NVIDIA driver: 1) proprietary, and 2) opensource. We detect the flavor when compiling gdrdrv based on the source code of the NVIDIA driver. Different flavors come with different features and restrictions:

• gdrdrv compiled with the opensource flavor will provide functionality and high performance on all platforms. However, you will not be able to load this gdrdrv driver when the proprietary NVIDIA driver is loaded.
• gdrdrv compiled with the proprietary flavor can always be loaded regardless of the flavor of NVIDIA driver you have loaded. However, it may have suboptimal performance on coherent platforms such as Grace-Hopper. Functionally, it will not work correctly on Intel CPUs with Linux kernel built with confidential compute (CC) support, i.e. CONFIG_ARCH_HAS_CC_PLATFORM=y, WHEN CC is enabled at runtime.

Tests

Execute provided tests:

…

NUMA effects

Depending on the platform architecture, like where the GPU are placed in the PCIe topology, performance may suffer if the processor which is driving the copy is not the one which is hosting the GPU, for example in a multi-socket server.

In the example below, GPU ID 0 is hosted by CPU socket 0. By explicitly playing with the OS process and memory affinity, it is possible to run the test onto the optimal processor:

…

or on the other socket:

…

Restrictions and known issues

GDRCopy works with regular CUDA device memory only, as returned by cudaMalloc. In particular, it does not work with CUDA managed memory.

gdr_pin_buffer() accepts any addresses returned by cudaMalloc and its family. In contrast, gdr_map() requires that the pinned address is aligned to the GPU page. Neither CUDA Runtime nor Driver APIs guarantees that GPU memory allocation functions return aligned addresses. Users are responsible for proper alignment of addresses passed to the library.

Two cudaMalloc'd memory regions may be contiguous. Users may call gdr_pin_buffer and gdr_map with address and size that extend across these two regions. This use case is not well-supported in GDRCopy. On rare occassions, users may experience 1.) an error in gdr_map, or 2.) low copy performance because gdr_map cannot provide write-combined mapping.

In some GPU driver versions, pinning the same GPU address multiple times consumes additional BAR1 space. This is because the space is not properly reused. If you encounter this issue, we suggest that you try the latest version of NVIDIA GPU driver.

On POWER9 where CPU and GPU are connected via NVLink, CUDA9.2 and GPU Driver v396.37 are the minimum requirements in order to achieve the full performance. GDRCopy works with ealier CUDA and GPU driver versions but the achievable bandwidth is substantially lower.

If gdrdrv is compiled with the proprietary flavor of NVIDIA driver, GDRCopy does not fully support Linux with the confidential computing (CC) configuration with Intel CPU. In particular, it does not functional if CONFIG_ARCH_HAS_CC_PLATFORM=y and CC is enabled at runtime. However, it works if CC is disabled or CONFIG_ARCH_HAS_CC_PLATFORM=n. This issue is not applied to AMD CPU. To avoid this issue, please compile and load gdrdrv with the opensource flavor of NVIDIA driver.

To allow the loading of unsupported 3rd party modules in SLE, set allow_unsupported_modules 1 in /etc/modprobe.d/unsupported-modules. After making this change, modules missing the "supported" flag, will be allowed to load.

Bug filing

For reporting issues you may be having using any of NVIDIA software or reporting suspected bugs we would recommend you use the bug filing system which is available to NVIDIA registered developers on the developer site.

If you are not a member you can sign up.

Once a member you can submit issues using this form. Be sure to select GPUDirect in the "Relevant Area" field.

You can later track their progress using the My Bugs link on the left of