#13416·dynamo

feat(trtllm): add a context-first adapter to the standalone decode sidecar

Author: tmonty12Created Aug 17, 2026Updated Sep 19, 2026
Labelsenhancementbackend::trtllmrouterdep:implementingepp

Outcome

Make the standalone decode sidecar orchestrate the supported raw TensorRT-LLM context-first prefill-to-generation handoff after EPP selects a disaggregated plan.

The sidecar sends a non-streaming context-only request to the selected context worker, uses the returned context-transfer parameters to start a generation-only request on its local generation worker, and returns only the generation response stream to Gateway. KV bytes transfer directly between TensorRT-LLM workers; neither EPP nor the sidecar carries them.

Scope

  • Version-gate the supported raw trtllm-serve context/generation request and response contract.
  • Receive and validate the selected context endpoint through the standalone disaggregated EPP-to-sidecar contract.
  • Mint one globally unique, contract-compliant disaggregation request ID and use it for both legs.
  • Send a context_only request with stream=false; validate and retain its prompt tokens, first generated token, and context-transfer parameters.
  • Send the corresponding generation_only request to the selected local generation worker and expose only that response stream to Gateway.
  • Preserve supported error, cancellation, and stream-termination behavior without leaking paired reservations.
  • Add focused HTTP protocol coverage for context-first handoff.

Out of scope

  • Generation-first routing.
  • Delegating to the built-in trtllm-serve disaggregated orchestrator.
  • KV-byte proxying through EPP or the sidecar.
  • Multi-EPP disaggregated replication and recovery.
  • End-to-end deployment validation and benchmarks.

Dependencies

  • Milestone 3: TensorRT-LLM Aggregated Routing.
  • ai-dynamo/dynamo#13405 — decode-sidecar request contract.
  • ai-dynamo/dynamo#13406 and ai-dynamo/dynamo#13407 — role-scoped worker catalogs and paired selection.

References

Tracking

  • DEP: ai-dynamo/dynamo#11661
  • Milestone: TensorRT-LLM Disaggregated Routing