[Bug] DistServe Proxy requests can leak Prefill scheduler metadata and OOM the Prefill engine

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

In LMDeploy v0.17.0 DistServe mode, normal external requests sent to the Proxy /v1/completions endpoint can leave orphaned Prefill scheduler sessions/sequences.

The issue appears to be caused by a mismatch between the session id used by the DistServe cache-free path and the actual internal Prefill scheduler session id. The free request misses the real Prefill session, logs invalid free, and leaves preserved Prefill scheduler metadata behind. Repeated normal requests make Prefill-side metadata and RSS grow continuously.This eventually killed the Prefill PyTorch ZMQ engine and made the DistServe service degraded/unavailable.

Detail

Analyzed version:

• LMDeploy v0.17.0
• Source snapshot used for analysis: lmdeploy-main-src/lmdeploy-main

Relevant source paths:

• lmdeploy/serve/proxy/proxy.py
• lmdeploy/serve/openai/endpoints/distserve.py
• lmdeploy/pytorch/engine/engine_instance.py
• lmdeploy/pytorch/engine/engine_loop.py
• lmdeploy/pytorch/disagg/conn/engine_conn.py
• lmdeploy/pytorch/paging/scheduler.py
• lmdeploy/pytorch/paging/seq_states/states.py

Mechanism:

1. In DistServe mode, Proxy splits a normal client request into a Prefill request and a Decode request.
2. Proxy sends the Prefill request with cache preservation enabled: with_cache=True and preserve_cache=True.
3. Proxy then builds the Decode migration_request with remote_session_id = int(prefill_info.get("id")).
4. After Decode-side migration, the migration loop sends a cache-free request back to Prefill using this remote_session_id.
5. Prefill handles that request through /distserve/free_cache / the DistServe connection handler and attempts to free scheduler.sessions[remote_session_id].
6. The id from the OpenAI-style Prefill response does not match the internal Prefill scheduler session id that owns the preserved sequence. The free path therefore misses the real session and logs invalid free.
7. Because the Prefill sequence was created with preserve_cache=True, the normal finish path preserves it for migration instead of removing all scheduler metadata immediately. Since the later free-cache request misses the real session, the Prefill-side SchedulerSession, SchedulerSequence, seq_manager._seq_map, and sequence history metadata remain orphaned.

In the reproduction below, Decode did not show the same residual sessions / seq_manager._seq_map growth. The accumulation was observed on the Prefill side.

Reproduction

Test environment:

• LMDeploy v0.17.0
• 1 Prefill + 1 Decode DistServe deployment
• Model: Qwen2.5-0.5B-Instruct
• Client traffic was sent to the normal Proxy /v1/completions endpoint.
• Prefill container memory and swap were limited to 4GiB.
• The 4GiB limit was used only to make the final availability impact observable in a bounded test run. The underlying issue is the monotonic accumulation of Prefill scheduler metadata; with a higher memory limit, the same growth pattern would require more requests before causing memory pressure or OOM.
• An audit-only read-only endpoint was added to collect internal scheduler counters.

Reproduction is simply to keep sending normal completion requests to the DistServe Proxy. In my run, the Prefill container was limited to 4GiB and the same service was kept running until the accumulated Prefill metadata caused OOM.

PROXY_URL=http://127.0.0.1:19000
MODEL=Qwen2.5-0.5B-Instruct

for i in $(seq 1 90000); do
  curl --silent --show-error --output /dev/null \
    -X POST "${PROXY_URL}/v1/completions" \
    -H 'Content-Type: application/json' \
    -d "{
      \"model\": \"${MODEL}\",
      \"prompt\": \"LC1 continuous normal request ${i}. Please answer briefly.\",
      \"temperature\": 0,
      \"max_tokens\": 1,
      \"stream\": false
    }"
done

Observed progression:

Additional observed counters:

Before OOM:
proxy_prefill_dispatch=87344
proxy_decode_dispatch=87344
prefill_invalid_free=86593
decode_migration_begin=86604
decode_migration_done=86604

The last cycle before OOM showed the Prefill container at 3.995GiB / 4GiB with 86417 residual Prefill sessions/sequences. In the next cycle, Docker reported the Prefill container as OOM-killed:

name=/lmdeploy0170-prefill
running=true
oom_killed=true
exit_code=0
memory_limit=4294967296
memory_swap=4294967296
status=running

After OOM, the container still appeared to be running, but the PyTorch ZMQ engine was dead.

Prefill health check:

{"status":"unhealthy","message":"PyTorch ZMQ engine process is not alive."}
http=503

Prefill audit RPC:

{"status":"ERROR","error_type":"RPCServerDeadError","error":"PyTorch ZMQ engine process is not alive."}
http=500

Proxy node status after OOM only showed the Decode node. The Prefill node disappeared:

{
  "http://127.0.0.1:19002": {
    "role": 3,
    "models": ["Qwen2.5-0.5B-Instruct"],
    "unfinished": 134,
    "speed": null
  }
}

Impact:

• Unbounded Prefill-side CPU metadata growth.
• Continuous growth of Prefill worker RSS and container memory usage.
• Accumulation of orphaned SchedulerSession, SchedulerSequence, seq_manager entries, and sequence history metadata.
• Under memory limits, the Prefill PyTorch ZMQ engine can be OOM-killed while the container still appears running.
• Prefill /health returns 503 and Proxy loses the Prefill node, causing service degradation or denial of service.

The trigger path is the normal external Proxy /v1/completions DistServe request path. It does not require malformed direct requests to internal DistServe APIs.

Environment

Target version: LMDeploy v0.17.0
Model: Qwen2.5-0.5B-Instruct
Python: 3.12.3 (main, Jul 15 2026, 23:46:41) [GCC 13.3.0]
CUDA available: True
GPU 0: NVIDIA A100 80GB PCIe
GPU 0 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 13.0, V13.0.88
CUDA Driver Version: 590.48.01
PyTorch: 2.13.0+cu130
sglang: 0.5.19
sglang-kernel: 0.4.6.post1
flashinfer_python: 0.6.18
flashinfer_cubin: 0.6.18
flashinfer_jit_cache: 0.6.18+cu130
triton: 3.7.1
transformers: 5.12.1
numpy: 2.3.5
aiohttp: 3.14.3
fastapi: 0.141.1
huggingface_hub: 1.30.0
interegular: 0.3.3
modelscope: 1.39.1
orjson: 3.12.0
outlines: 0.1.11
packaging: 26.3
psutil: 7.2.2
pydantic: 2.13.5
python-multipart: 0.0.32
pyzmq: 27.2.0
uvicorn: 0.52.4
uvloop: 0.22.1
xgrammar: 0.2.1
openai: 2.6.1
tiktoken: 0.14.0
torchcodec: 0.15.0+cu130
ulimit soft: 1024

Error traceback