#261·nano-vllm

[Bug] nano-vllm cannot start on Windows

Author: abdullahmoosaCreated Sep 1, 2026Updated Sep 1, 2026

Summary

On Windows, the engine crashes while starting up, before it even loads a model. This happens with a single GPU and default settings. Two separate things fail, both in nanovllm/engine/model_runner.py:

  1. nano-vllm asks PyTorch to use NCCL, the library used for communication between GPUs. PyTorch on Windows doesn't include NCCL, so this always fails.
  2. Once that first problem is worked around, startup fails again because a PyTorch networking component needs libuv, which is also missing from PyTorch on Windows.

So even a fully working Windows setup (NVIDIA GPU, correct drivers, all dependencies installed) cannot run nano-vllm at all.

Steps to reproduce

On Windows 11 with an NVIDIA GPU and Python 3.12, install the project and run:

bash
python example.py

First error:

RuntimeError: Distributed package doesn't have NCCL built in

After switching to a communication backend that exists on Windows (gloo), a second error appears:

RuntimeError: use_libuv was requested but PyTorch was build without libuv support

Root cause

  1. model_runner.py always requests the "nccl" backend, but NCCL only ships with PyTorch on Linux. This is the case even when there is just one GPU, where no GPU-to-GPU communication is actually needed.
  2. The networking part of PyTorch's startup defaults to libuv. Windows builds of PyTorch don't have it, but PyTorch does provide a setting (USE_LIBUV=0) that switches to older networking code that works on every platform.

Suggested fix

Two small changes, both in model_runner.py, with no effect on Linux:

  • Request NCCL only when it is available, and fall back to gloo otherwise. With a single GPU this changes nothing — the code only communicates between GPUs when tensor parallelism is turned on (tensor_parallel_size > 1).
  • On Windows, set USE_LIBUV=0 so PyTorch uses its cross-platform networking code.

I have both changes running locally and example.py works end to end on the hardware below. Happy to open a PR if you'd like Windows to work.

Expected outcome

  • Native Windows becomes a supported path: the engine starts and runs on a single GPU without WSL, exactly as on Linux today.
  • Linux behavior is unchanged — the code takes exactly the same path as before.
  • Multi-GPU tensor parallelism stays Linux-only, since Windows PyTorch has no NCCL at all. WSL remains the recommended route for that feature.

Environment

  • OS: Windows 11, NVIDIA driver 536.52 (CUDA 12.2)
  • GPU: RTX 4060 Laptop 8GB
  • Python 3.12.10 (venv; project requires Python < 3.13)
  • torch 2.6.0+cu124, transformers 5.16.1, triton-windows 3.2.0.post21, flash-attn 2.7.4.post1 (community Windows wheel), nano-vllm 0.2.0
  • Model: Qwen/Qwen3-0.6B from a local directory

Note: pip install . also fails on Windows because triton and flash-attn have no Windows versions on PyPI (see the closed #37 and #57). I used the triton-windows package and a community-built flash-attn wheel instead.

Source: GeeeekExplorer/nano-vllm