PP on docker: gloo control plane resolves to bridge IP (535ms per send) and presents as a long-context decode collapse — suggest defaulting GLOO_SOCKET_IFNAME=lo for same-netns ranks

Summary

On a single-container PP deployment, the cpu:gloo control plane (isend_object for per-step scheduler / speculative-decode feedback) resolves the container hostname to the docker bridge IP (172.17.0.2) instead of loopback. Every control send then costs ~535 ms (measured via torch-profiler user_annotation gloo:send). The payload grows with context (block tables / indices), so the tax presents as a long-context decode collapse that looks like a compute problem but isn't.

One-line fix:

-e GLOO_SOCKET_IFNAME=lo

Originally filed against the sm_80 backport fork as wtdcode/vllm-backport#111; the cpu:gloo group for PP coordination is created the same way in upstream parallel_state.py, so the failure mode applies to any single-container / docker-bridge PP deployment.

Environment

• 4× NVIDIA CMP 170HX (GA100, sm_80, 64 GiB, PCIe Gen2 x4, no P2P), single docker container, default bridge networking
• DeepSeek-V4.1-Flash (764B, EXL3 2/3 bpw mixed), PP4 (10,10,11,9), DSpark num_speculative_tokens=5, kv-cache-dtype fp8_ds_mla, 1M context, RustFrontend, prefix caching on
• backport image v0.13.0-lineage with exllamav3 1.5.3

Symptom

Single-stream decode collapsed with context while all silicon sat idle:

During deep-context decode, per-rank GPU busy was 8–19 % and every process (EngineCore, RustFrontend, API server, all 4 workers) sat at 0–3 % CPU — the pipeline was waiting on the control plane, not computing.

Root cause

Worker traces show the wait carrier directly:

user_annotation gloo:send   avg 535 ms/call, ~4.7 calls/s during decode

Same ~535 ms at 60K and 524K windows (constant per-send floor), but the number of bytes per send grows with context, which is what produces the context-dependent collapse. Gloo's default transport picks the interface from the container hostname → docker bridge, even though all ranks share a network namespace where loopback is the correct path.

Fix and results

-e GLOO_SOCKET_IFNAME=lo on the container (for multi-host deployments, set it to the real internal NIC instead — do not use lo across hosts).

Single-stream decode, warm, identical prompts:

Slope across 32K→524K: -37 % → -10 %. Cold prefill (fully distinct fillers, see caveat) also improved ~+55 % (524K TTFT 129 s ≈ 4,060 tok/s) since chunk control messages share the same group.

No regressions: quality suite 7/7 + vision 2/2 unchanged; concurrency C8 200.6 / C16 199.4 / C24 252.1 decode tok/s (mixed scenarios, warm round).

Suggestions

1. Consider defaulting GLOO_SOCKET_IFNAME=lo when all PP ranks live in one network namespace, or warning at startup when the gloo interface resolves to a docker-bridge subnet — the failure mode is silent and looks like a model/kernel problem.
2. Benchmarking caveat we hit while verifying (may save others time): prefix caching in this stack matches blocks by content, so a changing head-nonce does not prevent hits — honest cold-prefill benchmarks need fully distinct filler text per point.

Happy to provide the trace-analysis tooling (per-rank kernel pickles, gloo annotation comparison) if useful.