feat(trtllm): add a context-first adapter to the standalone decode sidecar
Author: tmonty12Created Aug 17, 2026Updated Sep 19, 2026
Labelsenhancementbackend::trtllmrouterdep:implementingepp
Outcome
Make the standalone decode sidecar orchestrate the supported raw TensorRT-LLM context-first prefill-to-generation handoff after EPP selects a disaggregated plan.
The sidecar sends a non-streaming context-only request to the selected context worker, uses the returned context-transfer parameters to start a generation-only request on its local generation worker, and returns only the generation response stream to Gateway. KV bytes transfer directly between TensorRT-LLM workers; neither EPP nor the sidecar carries them.
Scope
- Version-gate the supported raw
trtllm-servecontext/generation request and response contract. - Receive and validate the selected context endpoint through the standalone disaggregated EPP-to-sidecar contract.
- Mint one globally unique, contract-compliant disaggregation request ID and use it for both legs.
- Send a
context_onlyrequest withstream=false; validate and retain its prompt tokens, first generated token, and context-transfer parameters. - Send the corresponding
generation_onlyrequest to the selected local generation worker and expose only that response stream to Gateway. - Preserve supported error, cancellation, and stream-termination behavior without leaking paired reservations.
- Add focused HTTP protocol coverage for context-first handoff.
Out of scope
- Generation-first routing.
- Delegating to the built-in
trtllm-serve disaggregatedorchestrator. - KV-byte proxying through EPP or the sidecar.
- Multi-EPP disaggregated replication and recovery.
- End-to-end deployment validation and benchmarks.
Dependencies
- Milestone 3: TensorRT-LLM Aggregated Routing.
- ai-dynamo/dynamo#13405 — decode-sidecar request contract.
- ai-dynamo/dynamo#13406 and ai-dynamo/dynamo#13407 — role-scoped worker catalogs and paired selection.
References
Tracking
- DEP: ai-dynamo/dynamo#11661
- Milestone: TensorRT-LLM Disaggregated Routing
Source: ai-dynamo/dynamo