#5548·inference

BUG: stop vLLM inference when requests are aborted or disconnected

Author: amumu96Created Sep 17, 2026Updated Sep 17, 2026
Labelsbuggpu

Problem

Streaming vLLM requests can continue consuming GPU resources after the client disconnects. The current async-generator aclose() propagation is not deterministic across the REST, Xoscar, and vLLM boundaries. In addition, OpenAI-compatible request_id values are left inside generation config instead of being propagated to the model actor, so the existing abort endpoint returns NO_OP for vLLM.

Proposed change

  • Propagate one request ID from the REST API through the model actor into vLLM.
  • Track active vLLM request IDs and call the underlying engine abort API.
  • Explicitly abort SSE requests when the client disconnects.
  • Keep normal completion abort-free and preserve DONE, NO_OP, and NOT_FOUND outcomes.
  • Cover explicit abort, disconnect cleanup, normal completion, unsupported engines, duplicate aborts, and concurrent requests.

Validation expectations

  • Focused unit tests for REST request-ID propagation and vLLM lifecycle cleanup.
  • Real GPU validation showing inference stops after disconnect and explicit abort.