[Bug] nano-vllm cannot start on Windows
Summary
On Windows, the engine crashes while starting up, before it even loads a model. This happens with a single GPU and default settings. Two separate things fail, both in nanovllm/engine/model_runner.py:
- nano-vllm asks PyTorch to use NCCL, the library used for communication between GPUs. PyTorch on Windows doesn't include NCCL, so this always fails.
- Once that first problem is worked around, startup fails again because a PyTorch networking component needs libuv, which is also missing from PyTorch on Windows.
So even a fully working Windows setup (NVIDIA GPU, correct drivers, all dependencies installed) cannot run nano-vllm at all.
Steps to reproduce
On Windows 11 with an NVIDIA GPU and Python 3.12, install the project and run:
python example.pyFirst error:
RuntimeError: Distributed package doesn't have NCCL built inAfter switching to a communication backend that exists on Windows (gloo), a second error appears:
RuntimeError: use_libuv was requested but PyTorch was build without libuv supportRoot cause
model_runner.pyalways requests the "nccl" backend, but NCCL only ships with PyTorch on Linux. This is the case even when there is just one GPU, where no GPU-to-GPU communication is actually needed.- The networking part of PyTorch's startup defaults to libuv. Windows builds of PyTorch don't have it, but PyTorch does provide a setting (
USE_LIBUV=0) that switches to older networking code that works on every platform.
Suggested fix
Two small changes, both in model_runner.py, with no effect on Linux:
- Request NCCL only when it is available, and fall back to gloo otherwise. With a single GPU this changes nothing — the code only communicates between GPUs when tensor parallelism is turned on (
tensor_parallel_size > 1). - On Windows, set
USE_LIBUV=0so PyTorch uses its cross-platform networking code.
I have both changes running locally and example.py works end to end on the hardware below. Happy to open a PR if you'd like Windows to work.
Expected outcome
- Native Windows becomes a supported path: the engine starts and runs on a single GPU without WSL, exactly as on Linux today.
- Linux behavior is unchanged — the code takes exactly the same path as before.
- Multi-GPU tensor parallelism stays Linux-only, since Windows PyTorch has no NCCL at all. WSL remains the recommended route for that feature.
Environment
- OS: Windows 11, NVIDIA driver 536.52 (CUDA 12.2)
- GPU: RTX 4060 Laptop 8GB
- Python 3.12.10 (venv; project requires Python < 3.13)
- torch 2.6.0+cu124, transformers 5.16.1, triton-windows 3.2.0.post21, flash-attn 2.7.4.post1 (community Windows wheel), nano-vllm 0.2.0
- Model: Qwen/Qwen3-0.6B from a local directory
Note: pip install . also fails on Windows because triton and flash-attn have no Windows versions on PyPI (see the closed #37 and #57). I used the triton-windows package and a community-built flash-attn wheel instead.
Source: GeeeekExplorer/nano-vllm