uccl

## About UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., IBGDA), with two key focuses: * **Flexibility** for high performance in fast-evolving ML workloads * **Portability** for connecting heterogeneous GPUs in ML workloads An UCCL overview can be found in this [slide deck](https://docs.google.com/presentation/d/1LQxZzxghRmua4FkfQjWu69wXy9hrs9V_tXrXt_DT-F4/edit?usp=sharing) with the following components: * **[UCCL-collective](collective/)** (UCCL-Tran) serves as a drop-in replacement for NCCL/RCCL (e.g., requiring no changes to application code), and significantly outperforms them in both latency and throughput across various settings. UCCL-collective performance comparison * On six HGX servers (across two racks) with 8x400G CX-7 RoCE NICs and 8xH100 GPUs, UCCL-collective outperforms NCCL by up to **2.5x** for AllReduce:

* On two AWS `g4dn.8xlarge` instances with 1x50G ENA NICs and 1xT4 GPUs within the same cluster placement group, UCCL-collective outperforms NCCL by up to **3.7x** for AllReduce:

UCCL-collective high-level design * UCCL-collective aims to: * rearchitect the CCL layer (while keeping NCCL APIs) to unleash the full potential of network hardware * rearchitect the network transport layer to be fast and extensible * support heterogeneous GPU and networking vendors such as Nvidia, AMD, and Broadcom * become an open and collaborative platform for GPU communication research * UCCL-collective has built a fast and extensible transport layer in software, which has created many benefits. * For example, existing network transports under NCCL (i.e., kernel TCP and RDMA) leverage one or few network paths to stream huge data volumes, thus prone to congestion happening in datacenter networks. * Instead, UCCL-collective employs packet spraying in software to leverage abundant network paths to avoid "single-path-of-congestion". * More benefits include: 1) packet spraying with 256 paths, 2) advanced congestion control such as latency-based and receiver-driven ones, 3) efficient loss recovery by selective repeat, and 4) widely usable in public clouds with legacy NICs and Ethernet. Feel free to check out our full [technical report](https://arxiv.org/pdf/2504.17307). * **[UCCL-P2P](p2p/)** provides NIXL-style initiator-target transfer APIs. UCCL-P2P is purposely designed for the next-gen 800Gbps NICs with efficient multi-threaded transfer engines. UCCL-P2P performance comparison * Message transfer bandwidth over RDMA on AMD MI300X + Broadcom Thor-2:

* **[UCCL-EP](ep/)** allows running DeepEP atop of heterogeneous hardware platforms, including AMD and Nvidia GPUs, and any RDMA NICs such as AWS EFA NICs and Broadcom NICs, while achieving IBGDA-level performance. UCCL-EP performance comparison * EP32 dispatch and combine on AWS p5en (8x H200 + 16x 200Gb/s EFA):

UCCL has been adopted as part of the AMD [TheRock](https://github.com/ROCm/TheRock) ecosystem. ## Road Map More UCCL features are under development in this repo, currently including: - ✅ More efficient KV cache transfer engine (e.g., better Mooncake) - ✅ Supporting AMD GPUs - ✅ Supporting RDMA (NVIDIA, Broadcom), AWS EFA, GCP TCPX, TCP - ✅ Efficient and portable expert-parallel communication - ✅ Supporting all NIC vendors, including Nvidia, AWS EFA, and Broadcom - ✅ Supporting AMD GPUs - ✅ Better flow control to avoid congestion - ☐ Supporting other AI accelerators, such as TPUs and Trainium - Re-architecting NCCL to unleash network hardware performance - SM-efficient communication kernels - Fine-grained compute-communication overlapping - ☐ Device kernels in vendor-agnostic Triton language - Efficient consumer GPU communication - Faster collectives on 4090/5090/GB10 - Expert-parallel communication on 4090/5090/GB10 ## Quick Start The easiest way to use UCCL is to first build based on your platform. The build script will automatically detect the `py_version` of your current environment. If you need to compile UCCL for a specific python version, please specify the `py_version`, such as `3.10`. ```bash git clone https://github.com/uccl-project/uccl.git && cd uccl # Eg, bash build.sh cu12 ep --install bash build.sh [cu12|cu13|roc7|roc6|therock] [all|ccl_rdma|ccl_efa|p2p|ep] \ [py_version] [rocm_index_url] --install ``` > Note: > - By default, `build.sh cu12` targets CUDA 12.8 and `build.sh roc7` targets ROCm 7.1, but you can also specify `cu13|roc6` to target CUDA 13.0 or ROCm 6.4. > - UCCL uses [nanobind](https://github.com/wjakob/nanobind) for C++/Python bindings. On Python 3.12+, wheels are tagged `cp312-abi3` (stable ABI, one wheel for all 3.12+ interpreters); on older Pythons, wheels are CPython-version-specific. > - When building for ROCm with python packaging through TheRock, please specify your ROCm index url; the default is `https://rocm.prereleases.amd.com/whl/gfx94X-dcgpu` and it may not be what you want. When installing UCCL wheels for TheRock, please provide pip with the index url and add the optional extra `[rocm]` to the wheel, e.g., `pip install --extra-index-url https://rocm.prereleases.amd.com/whl/gfx94X-dcgpu wheelhouse-therock/uccl-0.0.1.post4-py3-none-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl[rocm]`. Then, when running your PyTorch applications, set the environment variable accordingly: ```bash # NCCL over IB/RoCE on x86 or GH200 ARM hosts NCCL_NET_PLUGIN=`python -c "import uccl; print(uccl.nccl_plugin_path())"` # RCCL over IB/RoCE on x86 hosts NCCL_NET_PLUGIN=`python -c "import uccl; print(uccl.rccl_plugin_path())"` # NCCL over AWS EFA NICs (p4d and p4de only) LD_PRELOAD=`python -c "import uccl; print(uccl.efa_nccl_path())"` NCCL_NET_PLUGIN=`python -c "import uccl; print(uccl.efa_plugin_path())"` ``` Now, you can just run your PyTorch applications and enjoy UCCL performance benefits! ## Dev Guide Click me First clone the UCCL repo and init submodules: ```bash git clone https://github.com/uccl-project/uccl.git export UCCL_HOME=$(pwd)/uccl ``` To build UCCL for development, you need to install some common dependencies: ``` … ``` **Alternatively, use [uv](https://docs.astral.sh/uv/) for a faster, conda-free dev setup:** ```bash source scripts/bootstrap.sh ``` This single command installs `uv` if missing, creates a `.venv` virtualenv with Python 3.12, pins all non-CUDA dev tools (`black`, `clang-format`, `pytest`, `paramiko`, etc.) from `uv.lock` via `uv sync --group dev`, and then runs `ep/install_deps.sh` to install CUDA-specific packages (torch, etc.) with automatic hardware detection. For quick installation with docker, you can directly dive into: * [`UCCL-Collective RDMA`](collective/rdma/README.md): Collectives for Nvidia/AMD GPUs + IB/RoCE RDMA NICs (currently support Nvidia and Broadcom NICs) * [`UCCL-Collective EFA`](collective/efa/README.md): Collectives for AWS EFA NIC (currently support p4d.24xlarge) > On p5/p5e/p5en/p6, the offical [aws-ofi-nccl](https://github.com/aws/aws-ofi-nccl) NCCL plugin with proper [env variables](https://github.com/uccl-project/uccl/blob/deeeaa36ebe5440449273633652d2b7d77f4a7aa/collective/efa/run_nccl_test.sh#L76-L77) already makes NCCL perform excellent * [`UCCL-Collective AFXDP`](collective/afxdp/README.md): Collectives for Non-RDMA NICs (currently support AWS ENA NICs and IBM VirtIO NICs) * [`UCCL-P2P`](p2p/README.md): P2P for RDMA NICs and GPU IPCs (currently support Nvidia/AMD GPUs and Nvidia/Broadcom NICs) * [`UCCL-EP`](ep/README.md): EP for MoE training and inference with DeepEP-compatible APIs (currently support Nvidia/AMD GPUs and Nvidia/Broadcom/EFA NICs) ## Adoptions * NVIDIA NeMo agent framework integrates UCCL-EP for expert-parallel communication: [homepage](https://docs.nvidia.com/nemo/automodel/latest/apidocs/nemo_automodel/nemo_automodel.components.moe.uccl_ep.html). * NVIDIA NIXL inference transfer library integrates UCCL-P2P as a RDMA backend: [release page](https://github.com/ai-dynamo/nixl/releases/tag/0.9.0). * Red Hat/IBM/Google llm-d distributed inference stack leverages UCCL-P2P for KV-cache transfer: [blog](https://llm-d.ai/blog/llm-d-v0.5-sustaining-performance-at-scale). * AMD Primus training framework uses UCCL-EP for expert-parallel communication: [code](https://github.com/AMD-AGI/Primus/tree/main/examples/moe_package). * AMD TheRock build platform incorporates UCCL-Tran, UCCL-EP, and UCCL-P2P: [homepage](https://github.com/ROCm/TheRock/tree/main/external-builds/uccl). ## Citation The code in this repository is mostly described in the papers below. Please consider citing this work if you find the repository helpful. ```bibtex @article{uccl_tran, title={UCCL-Tran: An Extensible Software Transport Layer for GPU Networking}, author={Zhou, Yang and Chen, Zhongjie and Mao, Ziming and Lao, ChonLam and Yang, Shuo and Kannan, Pravein Govindan and Gao, Jiaqi and Zhao, Yilong and Wu, Yongji and You, Kaichao and Ren, Fengyuan and Xu, Zhiying and Raiciu, Costin and Stoica, Ion}, journal={USENIX OSDI}, year={2026} } ``` ```bibtex @article{uccl_ep, title={UCCL-EP: Portable Expert-Parallel Communication}, author={Mao, Ziming and Zhang, Yihan and Cui, Chihan and You, Kaichao and Chen, Zhongjie and Xu, Zhiying and Shenker, Scott and Raiciu, Costin and Zhou, Yang and Stoica, Ion}, journal={USENIX OSDI}, year={2026} } ``` ## Acknowledgement UCCL is being actively developed at [UC Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/) and [UC Davis ArtSy lab](https://github.com/artsy-lab). We enthusiastically welcome open-source developers joining us! UCCL is generously supported by (in alphabetical order): [AMD](https://www.amd.com/en.html), [AWS](https://aws.amazon.com/), [Broadcom](https://www.broadcom.com/), [CloudLab](https://www.cloudlab.us/), [Google Cloud](https://cloud.google.com/), [IBM](https://www.ibm.com/), [Lambda](https://lambda.ai/), [Mibura](https://www.mibura.com/).