#231·GLM-OCR

vLLM serve fails on WSL2 — EngineCore segfault during multimodal encoder initialization

Author: sfingaliCreated Jul 25, 2026Updated Jul 26, 2026

Environment
- OS: Windows 10 + WSL2 (Ubuntu 24.04)
- GPU: NVIDIA RTX 3090 (24 GB, sm_86)
- Driver: 596.x (WSL2)
- vLLM: 0.26.0
- Python: 3.10
- Model: zai-org/GLM-OCR (BF16, 2.47 GB)
Steps

 VLLM_WSL2_ENABLE_PIN_MEMORY=1 vllm serve zai-org/GLM-OCR --host 0.0.0.0 --port 8080                                                                      
                                                                                                                                                          
 Observed behavior                                                                                                                                        
 Crashes during encoder cache initialization with EngineCore failure. Multiple crash modes across attempts:                                               
 1. --enforce-eager OFF: dies at CUDA graph capture → flashinfer JIT needs nvcc (not in WSL2)                                                             
 2. --enforce-eager ON: model loads, weights loaded, then crashes at warmup — Triton M-RoPE error in glm4.py:139                                          
 3. All paths end at RuntimeError: Engine core initialization failed. Failed core proc(s): {}                                                             
 What works                                                                                                                                               
 - Native Linux: same command works (per vLLM issue #29482)                                                                                               
 - HuggingFace transformers (pure Python, no vLLM): works on WSL2 — model loads, CogViT processes images correctly                                        
 - GGUF via Ollama: model loads but produces 0-3 chars output (CogViT visual encoder quantization collapse)                                               
 Root cause analysis                                                                                                                                      
 vLLM's V1 engine multimodal encoder path triggers torch.compile/torch.dynamo on the CogViT encoder. This fails in WSL2's GPU paravirtualization          
 layer (dxgkrnl). The pinned memory workaround (VLLM_WSL2_ENABLE_PIN_MEMORY=1, PR #41496) was built for text-only LLMs — the vision encoder path is       
 untested on WSL2.                                                                                                                                        
                                                                                                                                                          
 This mirrors vLLM issue #29482 (DeepSeek-OCR segfault on WSL2, closed as not_planned).                                                                   
 Request                                                                                                                                                  
 Document WSL2 as unsupported for vLLM inference, or provide a WSL2-specific encoder path that bypasses torch.compile on the visual encoder.