candle-vllm

Efficient, easy-to-use platform for inference and serving local LLMs including an OpenAI compatible API server.

English | 简体中文

✨ Why Candle-vLLM?

Quick Start

Install

Option 1 — One-line install (DEB or binary)

curl -sSL https://ericlbuehler.github.io/candle-vllm/install.sh | bash

Option 2 — Build from source

git clone [email protected]:EricLBuehler/candle-vllm.git
cd candle-vllm

# CUDA (11+, 12+, 13.0) — remove flashinfer,cutlass for sm_70/sm_75
cargo install --features cuda,nccl,flashinfer,cutlass --path .

# macOS/Metal
cargo install --features metal --path .

Option 3 — Docker

# Pass custom SM version and CUDA version: ./build_docker.sh "cuda,nccl,flashinfer,cutlass" sm_90 13.0.0
./build_docker.sh "cuda,nccl,flashinfer,cutlass"

▶️ Run

Using HuggingFace Model ID:

candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

candle-vllm --m zai-org/GLM-5.2-FP8 --d 0,1,2,3,4,5,6,7 --ui-server

Using local model path:

# Local safetensors directory
candle-vllm --d 0,1,2,3,4,5,6,7 --m /home/data/GLM-5.2-FP8/ --ui-server

# Local GGUF file (single or split-shard)
candle-vllm --d 0,1 --m /home/data/model-Q4_K_M.gguf --ui-server

# Local directory containing GGUF files (auto-detected)
candle-vllm --d 0,1 --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

Tip: Add --ui-server to launch the built-in ChatGPT-style Web UI. The UI server uses the API port minus one (e.g., API on 2000, UI on 1999).

Performance

Single-request decode speed (input 4k, output 1k, on Hopper 80G)

Demo Video — GPU & Apple Silicon

Chat demo on GPU (A100, BF16, QWen3-8B Reasoning Model)

Chat demo on Apple Silicon (M4, 16GB unified memory, Q2K, QWen3-8B)

Features

• OpenAI compatible API server for serving LLMs
• Streaming support in generation
• Efficient KV cache management with PagedAttention
• Continuous batching (batched decoding for incoming requests over time)
• In-situ quantization (and In-situ Marlin format conversion)
• GPTQ/Marlin format quantization (4-bit)
• Support Mac/Metal devices
• Support Multi-GPU inference (both multi-process and multi-threaded mode)
• Support Multi-node inference via TCP-based coordination
• Support Chunked Prefilling (default chunk size 8K)
• Support CUDA Graph
• Support Qwen3.5 MTP speculative decoding with CUDA Graph via --mtp
• Support external DFlash2 speculative decoding via --draft-model and --num-speculative-tokens
• Support Model Context Protocol (MCP) and OpenAI-compatible tool calling
• Support Prefix Caching
• Support Block-wise FP8 Models (SM90+, Qwen3 Series)
• Support FP8 KV Cache on all CUDA and Metal platforms
• Support TurboQuant KV Cache (turbo8/turbo4/turbo3) with native flash attention kernels
• Support Flashinfer Backend
• Support manual YaRN RoPE scaling override via --yarn-scaling-factor
• Support MXFP4/NVFP4 models
• Support GPTQ/AWQ/Marlin models, including pack-quantized AWQ MoE
• Support DeepSeek V3.2 and GLM-5.2 FP8 models

Usage

Running Models

Tip: By default, candle-vllm starts an OpenAI-compatible API server at http://localhost:2000. Add --ui-server to also launch the built-in ChatGPT-style Web UI.

…

FP8 / FP4 models

…

GGUF models

…

Multi-shard GGUF: Split GGUF files (e.g., model-00001-of-00005.gguf) are automatically discovered — both locally (from the same directory) and remotely (from the HuggingFace repo). When --f is a subfolder name (not ending in .gguf), all GGUF files in that remote subfolder are downloaded. Vision tower auxiliary files (mmproj*.gguf) are loaded on demand for multimodal models.

ISQ (In-situ quantization)

Simply add --isq parameter when running unquantized models:

candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

Options: q4_0, q4_1, q5_0, q5_1, q8_0, q2k, q3k, q4k, q5k, q6k

GPTQ / AWQ / Marlin models

…

️ TurboQuant KV Cache

TurboQuant compresses the KV cache using Walsh-Hadamard transform for higher throughput and longer context:

# Turbo4 (4-bit KV cache, ~3.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo4

# Turbo8 (FP8 K + 4-bit V, ~2.6x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo8

# Turbo3 (3-bit K + 4-bit V, ~4.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo3

# FP8 KV Cache
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --kvcache-dtype fp8

Note: TurboQuant uses native flash attention kernels (flashinfer is automatically disabled). Supported on both CUDA (SM70+) and Metal (Apple Silicon) platforms. MLA models (DeepSeek, GLM4/GLM-5.2) auto-fallback to standard KV cache as TurboQuant is incompatible with their compressed KV layout.

️ Multi-GPU Inference

Multi-process mode (recommended)

# QwQ-32B BF16 on two GPUs
candle-vllm --d 0,1 --w /home/QwQ-32B/

# QwQ-32B 4-bit AWQ on two GPUs
python3 examples/convert_awq_marlin.py --src /home/QwQ-32B-AWQ/ --dst /home/QwQ-32B-AWQ-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0,1 --w /home/QwQ-32B-AWQ-Marlin/

Note: Number of GPUs (--d) must be a power of 2 (e.g., 2, 4, or 8).

Multi-threaded mode (debug)

# Add --multithread parameter
candle-vllm --multithread --d 0,1 --w /home/QwQ-32B/

# Troubleshooting
export NCCL_P2P_DISABLE=1  # disable P2P if encountering illegal memory access

Speculative decoding: MTP and DFlash2

Candle-vLLM supports two speculative-decoding modes for Qwen3.5/3.6/3.8-family targets:

• Built-in MTP (Multi-Token Prediction): the target checkpoint contains mtp.* weights. Use --mtp N or the equivalent --num-speculative-tokens N.
• External DFlash2: use a matching DFlash2 draft checkpoint with --draft-model. If --draft-model is set, DFlash2 takes priority over MTP; --num-speculative-tokens optionally overrides the draft width (otherwise block_size - 1 from the draft config is used).

MTP example:

# Three MTP draft tokens per decode step
candle-vllm --m /data/Qwen3.5-35B-A3B-FP8/ \
  --num-speculative-tokens 3 --ui-server

DFlash2 example:

# Qwen3.8 target and its matching DFlash2 draft checkpoint
candle-vllm --m /data/Qwen3.8-27B-FP8/ \
  --draft-model /data/Qwen3.8-27B-DFlash2-FP8/ \
  --num-speculative-tokens 7 --ui-server

Multi-Node Inference

Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.

# On master node (192.168.1.100):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --master-port 29500

# On worker node (192.168.1.101):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100 --master-port 29500

All nodes must have model weights locally and be TCP-reachable on --master-port (default 29500).

NUMA Binding

Show command

sudo apt-get install numactl

# 8 GPUs, 2 NUMA nodes
MAP_NUMA_NODE=0,0,0,0,1,1,1,1 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3,4,5,6,7 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

# 4 GPUs
MAP_NUMA_NODE=0,0,0,0 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

numactl --cpunodebind=0 --membind=0 specifies the master rank's NUMA binding and must match MAP_NUMA_NODE.

⚙️ CLI Reference