DEP: EPP Embedded SelectionService Interface

Table of Contents

• Summary
• Motivation
• Proposal
  • Design Inputs
  • Component Boundary
  • Motivation
  • Goals
  • Worker Discovery
  • In-Process Interface
  • Aggregated Request Lifecycle
  • Tokenization
    • Problem with the Dynamo Preprocessor
    • Initial Implementation: External Tokenizer Service
      • Backend Tokenization Services
    • Long-Term: Tokens-In/Tokens-Out Execution
  • Disaggregated Routing
    • Overview
    • Plane 1: EPP Selection and Accounting
    • Plane 2: Decode Sidecar Execution
    • Backend Execution Contracts
      • vLLM
      • SGLang
      • TensorRT-LLM
  • Multi-Replica EPP State Sharing
    • Peer Discovery
    • State-Sharing Planes
  • EPP Embedded SelectionService vs Full Dynamo Runtime Router
    • Functional Gaps
• Implementation Milestones
  • Milestone 1: vLLM Aggregated Routing
  • Milestone 2: SGLang Aggregated Routing
  • Milestone 3: TensorRT-LLM Aggregated Routing
  • Milestone 4: Multi-EPP Aggregated Routing
  • Milestone 5: vLLM Disaggregated Routing
  • Milestone 6: SGLang Disaggregated Routing
  • Milestone 7: TensorRT-LLM Disaggregated Routing
  • Milestone 8: Multi-EPP Disaggregated Routing and Hardening
  • Milestone 9: Embedded EPP Observability
  • Functional-gap completion tasks
• Alternate Solutions
• Requirements
• Appendix
  • Environment Contract
    • EPP Request Plane
    • EPP Selection and Worker Metadata
    • Multi-Replica EPP State Sharing
  • Kubernetes Resource Model
  • Worker Catalog Field-to-EPP Mapping
• References

Summary

Embed SelectionService in EPP for runtime-free Dynamo KV-aware and load-aware routing.

Motivation

Preserve existing Gateway API and raw engine worker deployments while adding Dynamo selection, KV indexing, queueing, and accounting.

Proposal

This document outlines how Dynamo Endpoint Picker Plugin (EPP) embeds SelectionService to achieve a Dynamo runtime-free pathway that can integrate directly with raw engine workers. EPP handles the Envoy protocol, request normalization, and worker discovery, while SelectionService owns worker selection, KV indexing, scheduling, active-load accounting, and reservation state.

Design Inputs

• Standalone Selection Service is the API and behavior reference for SelectionService. It documents the standalone HTTP packaging and the embedded Rust API through SelectionServiceBuilder.
• DEP: Runtime-Free Router and Gateway On-Ramp is the broader migration proposal. This document narrows that proposal to the EPP-owned Gateway/Kubernetes adapter and its in-process, non-runtime interface to SelectionService.

Component Boundary

Motivation

Many users already run vllm serve or other OpenAI-compatible backends behind a Kubernetes Gateway in their own cluster. They may want Dynamo's KV-aware and load-aware routing, but do not want to adopt the full Dynamo runtime, Dynamo operator, DynamoGraphDeployment, discovery plane, or event plane.

The embedded EPP path gives those users an incremental adoption model. They keep their existing Gateway API resources and backend pods, add the Dynamo EPP, and let the EPP register eligible workers with its in-process SelectionService. The EPP remains the Gateway request-plane integration point, while SelectionService provides routing decisions and reservation accounting through a runtime-free in-process interface.

Goals

• Support a user-managed Kubernetes deployment where raw vLLM, SGLang, or TRTLLM pods sit behind Gateway API and GAIE.
• Let users add Dynamo routing without requiring the Dynamo runtime, Dynamo operator or DynamoGraphDeployment
• Keep Gateway API resources, EPP resources, and worker resources user-managed in the operator-free path.
• Keep routing policy, KV indexing, scheduling, and reservation accounting inside SelectionService instead of reimplementing those behaviors within the EPP.

Worker Discovery

The embedded EPP populates the SelectionService worker catalog through in-process method calls:

sequenceDiagram
    participant K as Kubernetes API
    participant E as EPP
    participant S as SelectionService
    participant W as Worker pod

    E->>K: Watch InferencePool
    E->>K: Watch pods matching selector.matchLabels
    K-->>E: Ready pod with pod IP
    E->>E: Build WorkerRequest
    E->>S: SelectionService::upsert_worker
    S->>W: Subscribe to KV events when configured

    K-->>E: Pod NotReady, terminating, deleted, or changed
    E->>S: SelectionService::delete_worker

See Appendix: Worker Catalog Field-to-EPP Mapping.

In-Process Interface

This design uses the embedded Rust API for SelectionService: the EPP constructs SelectionService with SelectionServiceBuilder and calls it directly.

Worker discovery uses the SelectionService worker-catalog methods:

The EPP uses the selection and lifecycle methods:

Aggregated Request Lifecycle

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant E as EPP
    participant T as Tokenizer sidecar
    participant S as SelectionService
    participant W as Worker

    C->>G: OpenAI request
    G->>E: ext_proc request headers/body
    E->>T: Render/tokenize request
    T-->>E: Token IDs
    E->>E: Apply candidate subset and mint reservation UUID
    E->>S: SelectionService::select_and_reserve
    S-->>E: worker_id, dp_rank, endpoint, overlap
    E-->>G: destination endpoint and routing headers
    G->>W: Forward request
    W-->>G: Response body starts
    G->>E: first non-empty response body
    E->>S: SelectionService::prefill_complete
    G-->>C: Stream response
    G->>E: response complete, cancel, or error
    E->>S: SelectionService::free_reservation

On successful selection, the EPP returns:

• x-gateway-destination-endpoint = podIP:port

Tokenization

SelectionService requires token IDs to construct block hashes and generate KV-overlap scores. It also indexes KV events emitted by the worker, which describe the tokens actually executed by that worker.

Problem with the Dynamo Preprocessor

Using the Dynamo OpenAI Preprocessor in EPP would create a separate tokenization path from the inference engine. The token sequence used for selection could differ from the token sequence in the worker's KV events, making KV-aware routing inaccurate.

Initial Implementation: External Tokenizer Service

The initial EPP implementation uses an external tokenizer service associated with the inference backend:

1. EPP sends the original OpenAI request to the tokenizer service.
2. The service returns engine-native token IDs.
3. EPP performs KV-aware selection with those IDs.
4. EPP forwards the original request unchanged to the selected worker.

Pros

• Uses the backend's tokenization path for routing.
• Keeps the selection layer independent of tokenizer libraries and model artifacts.
• Preserves the raw OpenAI request for execution.

Cons

• Adds a tokenizer RPC and its availability dependency to the request path.
• The worker tokenizes the request again for execution.
• Requires backend- and version-specific tokenizer-service configuration.

Backend Tokenization Services

• vLLM: The vLLM Render service provides rendered prompt token IDs through its chat-completions render endpoint. It can run separately from the model-serving worker.
• SGLang: SGLang provides a /tokenize endpoint that can serve as the backend tokenizer service. Its accepted request format and chat-template support must be version-gated before relying on it for KV-aware routing.
• TensorRT-LLM: TensorRT-LLM does not currently provide an equivalent tokenizer-service endpoint.

Long-Term: Tokens-In/Tokens-Out Execution

A backend-native tokens-in/tokens-out execution interface would eliminate the second tokenization step. EPP would obtain canonical input IDs from either Dynamo preprocessing or a tokenizer service, send those IDs to the selected worker, and receive output IDs for response handling.

• vLLM: POST /inference/v1/generate accepts token_ids; with detokenize=false, the response exposes generated token IDs.
• SGLang: POST /generate accepts token-ID input (input_ids). Adoption requires validating that the deployed version returns native output token IDs in the required response path.
• TensorRT-LLM: No corresponding EPP-compatible tokens-in/tokens-out interface is currently identified.

Disaggregated Routing

Overview

Disaggregated routing has two independent responsibilities. First, the EPP selects and books a compatible prefill/decode pair using two role-scoped SelectionService instances. Second, Gateway forwards the original request to the selected decode-sidecar endpoint, and that sidecar executes the backend-specific prefill/decode protocol using the selected prefill endpoint as a routing hint.

sequenceDiagram
    participant G as Gateway
    participant E as EPP
    participant PS as Prefill SelectionService
    participant DS as Decode SelectionService
    participant DX as Decode sidecar
    participant P as Prefill engine
    participant D as Local decode engine

    G->>E: ext_proc request headers and body
    E->>E: Tokenize for routing and derive role constraints
    par Book prefill leg
        E->>PS: select_and_reserve(prefill reservation)
        PS-->>E: prefill endpoint
    and Book decode leg
        E->>DS: select_and_reserve(decode reservation)
        DS-->>E: decode sidecar endpoint
    end
    E->>E: Commit plan only if both legs succeed
    E-->>G: x-gateway-destination-endpoint + prefill routing hint
    G->>DX: Original OpenAI request
    DX->>P: Backend-specific prefill request
    DX->>D: Backend-specific decode request
    P-->>D: KV transfer over backend transport
    D-->>DX: Response stream
    DX-->>G: Response stream
    G-->>E: Response progress and completion callbacks
    E->>PS: Release prefill reservation at handoff
    E->>DS: Release decode reservation at completion/cancel/error

Plane 1: EPP Selection and Accounting

The EPP constructs two embedded SelectionService instances for the same model: a prefill selector and a decode selector. Each selector receives only role-appropriate workers, KV-event inputs, policy configuration, and queue configuration. The EPP assigns a single logical plan ID plus role-qualified reservation IDs, for example (plan_id, prefill) and (plan_id, decode), and stores the mapping until the request terminates.

This mirrors the full Dynamo prefill/decode router more closely than a single shared selector. Prefill and decode have different queueing, scoring, and slot-accounting semantics, so each role gets an independent scheduler/policy profile and independent reservation state. The EPP-owned plan ledger ties the two role reservations together and translates lifecycle events to the correct selector.

Role-specific slot tracking:

Queuing is role-local. A prefill request waits only behind other prefill work in the prefill selector, and a decode request waits only behind decode work in the decode selector. The EPP should start both selection calls concurrently so one role is not systematically booked first while the other waits. No routing result is returned to Gateway until both role reservations succeed.

The transaction semantics are intentionally light. The two SelectionService calls are a lease-backed saga, not an atomic cross-service commit. If either leg fails, times out, loses its endpoint, or the client request is cancelled before dispatch, the EPP cancels any pending selection and frees any completed reservation. Leases or expiry should bound orphaned reservations if the EPP crashes after one role books but before cleanup. This guarantees no partial plan is exposed to Gateway, while accepting transient partial booked load inside one role selector during compensation.

sequenceDiagram
    participant E as EPP
    participant PS as Prefill SelectionService
    participant DS as Decode SelectionService
    participant G as Gateway

    E->>E: Mint plan_id and role-qualified reservation IDs
    par Prefill admission
        E->>PS: select_and_reserve(plan_id/prefill)
    and Decode admission
        E->>DS: select_and_reserve(plan_id/decode)
    end
    alt Both reservations succeed
        E->>E: Store plan ledger
        E-->>G: Decode destination and prefill hint
    else One leg fails or request is cancelled
        E->>PS: cancel/free prefill reservation if present
        E->>DS: cancel/free decode reservation if present
        E-->>G: Error or fallback
    end

Plane 2: Decode Sidecar Execution

After selection succeeds, the EPP returns x-gateway-destination-endpoint for the selected decode-sidecar endpoint and provides selected prefill metadata, such as x-prefiller-host-port, as orchestration metadata. The OpenAI request body remains the original client request; backend-specific disaggregation fields are not written by the EPP.

Gateway forwards the request to the decode sidecar. The sidecar validates the x-prefiller-host-port header format, strips or sanitizes orchestration headers before sending requests to engines, and performs the backend-specific P/D protocol. The sidecar owns prefill dispatch, decode dispatch, KV-transfer handoff coordination, cancellation cleanup, and the final response stream back to Gateway.

For the MVP, a syntactically valid x-prefiller-host-port is treated as authoritative. InferencePool membership validation and endpoint allowlisting are deferred as follow-up hardening.

Generic sidecar flow:

sequenceDiagram
    participant G as Gateway
    participant DX as Decode sidecar
    participant P as Prefill engine
    participant D as Local decode engine

    G->>DX: Original OpenAI request + selected prefill hint
    DX->>DX: Validate selected prefill header format
    DX->>DX: Strip client-supplied orchestration headers
    DX->>P: Backend-specific prefill leg
    DX->>D: Backend-specific decode leg
    P-->>D: Engine connector transfers KV state
    D-->>DX: Backend response stream
    DX-->>G: Client-facing response stream

Backend Execution Contracts

The decode sidecar receives the original OpenAI request and