vLLM serve fails on WSL2 — EngineCore segfault during multimodal encoder initialization
Author: sfingaliCreated Jul 25, 2026Updated Jul 26, 2026
Environment
- OS: Windows 10 + WSL2 (Ubuntu 24.04)
- GPU: NVIDIA RTX 3090 (24 GB, sm_86)
- Driver: 596.x (WSL2)
- vLLM: 0.26.0
- Python: 3.10
- Model: zai-org/GLM-OCR (BF16, 2.47 GB)
Steps
VLLM_WSL2_ENABLE_PIN_MEMORY=1 vllm serve zai-org/GLM-OCR --host 0.0.0.0 --port 8080
Observed behavior
Crashes during encoder cache initialization with EngineCore failure. Multiple crash modes across attempts:
1. --enforce-eager OFF: dies at CUDA graph capture → flashinfer JIT needs nvcc (not in WSL2)
2. --enforce-eager ON: model loads, weights loaded, then crashes at warmup — Triton M-RoPE error in glm4.py:139
3. All paths end at RuntimeError: Engine core initialization failed. Failed core proc(s): {}
What works
- Native Linux: same command works (per vLLM issue #29482)
- HuggingFace transformers (pure Python, no vLLM): works on WSL2 — model loads, CogViT processes images correctly
- GGUF via Ollama: model loads but produces 0-3 chars output (CogViT visual encoder quantization collapse)
Root cause analysis
vLLM's V1 engine multimodal encoder path triggers torch.compile/torch.dynamo on the CogViT encoder. This fails in WSL2's GPU paravirtualization
layer (dxgkrnl). The pinned memory workaround (VLLM_WSL2_ENABLE_PIN_MEMORY=1, PR #41496) was built for text-only LLMs — the vision encoder path is
untested on WSL2.
This mirrors vLLM issue #29482 (DeepSeek-OCR segfault on WSL2, closed as not_planned).
Request
Document WSL2 as unsupported for vLLM inference, or provide a WSL2-specific encoder path that bypasses torch.compile on the visual encoder.Source: zai-org/GLM-OCR