DEP: Native SGLang generate through Dynamo HTTP routing

Summary

Expose SGLang's native HTTP /generate through Dynamo using the same basic approach as the regular SGLang HTTP router: select workers, forward the native request, return the engine's JSON or SSE, and track load for the lifetime of the HTTP request. Use stock SGLang without engine changes.

Motivation

The existing token adapter requires token input, one sample, and incremental streaming. Native clients also need text inputs, batches, sampling options, extensions, and stream=false. Forwarding the native HTTP API lets SGLang retain ownership of generation validation and response formatting while Dynamo supplies its existing routing policies.

Proposal

flowchart LR
    Client --> Dynamo[Dynamo frontend and worker selection]
    Dynamo --> Sidecar[Selected sidecar]
    Sidecar --> Engine[Stock SGLang /generate]
    Engine -->|Native JSON or SSE| Sidecar
    Sidecar --> Dynamo
    Dynamo --> Client
    Dynamo -. P/D .-> Prefill[Selected prefill sidecar and engine]
    Prefill -. KV transfer .-> Engine
    Prefill -. Prompt logprobs .-> Dynamo

1. Read only the request fields needed for routing. Reuse admitted WorkerSets and existing KV-aware or load-based selection. Forward engine-specific fields without rebuilding an engine request schema. Use matching tokenization for KV routing; inputs without compatible routing tokens use a load-based policy.
2. Add reusable HTTP transport in the shared sidecar infrastructure, with method, endpoint path, headers, body bytes, and streamed response frames. Each backend explicitly registers the paths it exposes. Keep /generate routing projection, P/D coordination, and logprob merging in SGLang-specific modules; other endpoints can use the bridge without depending on SGLang. Forward status, end-to-end headers, and body chunks with backpressure and disconnect propagation. Unary and streaming generation use the same path. Preserve engine errors and do not silently fall back to the narrower token adapter.
3. Track in-flight HTTP work using the existing routing accounting. Keep selection/admission and load state in SelectionCore and the scheduler; keep reservation ownership, renewal and cleanup in RoutingHost, shared with the token path. The generic HTTP response wrapper holds the reservation. Release it on completion, failure, or disconnect. Do not add engine lifecycle events, cleanup acknowledgements, per-token observation, or per-sample scheduler accounting.
4. For P/D, reuse prefill/decode selection, insert bootstrap information and the selected DP ranks, and dispatch both HTTP requests concurrently. Preserve the caller's stream setting on both stages. SGLang performs KV transfer. Return the decode response; for scalar unary requests with logprobs and JSON prefill, merge meta_info.input_token_logprobs as the regular SGLang HTTP router does. Batch and streaming responses pass through without prompt-logprob reconstruction; no SSE parser is part of the implementation. This limited merge is the only generation-response interpretation required by the design.

Scope

• The compatibility reference is the regular SGLang HTTP router on the same stock engine version. Engine limitations shared with that router are outside this work.
• No SGLang patches, session coordination, new lifecycle protocol, richer P/D metadata handoff, or attempt to recreate full aggregated-engine output semantics.
• Use one load reservation per HTTP request per stage. Route a batch using its first prompt; leave sample and beam expansion to the engine. This is approximate routing load, not per-child scheduler accounting.
• Retain Dynamo's routing policies and discovery. Do not duplicate the entire SGLang router control plane. The initial transport does not retry generation and preserves native engine errors; differences from SMG retries and error wrapping must be documented and tested.
• Keep the feature opt-in through DYN_SGLANG_ENABLE_GENERATE and the sidecar's native HTTP option. Existing workers retain their current adapter.

Validation

Run identical native requests through the reference SGLang HTTP router and Dynamo on the same stock SGLang version. Cover unary/SSE, text/token inputs, batches, multiple samples, logprob options, independent P/D DP ranks, engine errors, disconnects, and partial-stage failures. Verify byte forwarding where no merge is needed, bounded response buffering, and HTTP load release. Fix Dynamo-only incompatibilities; record shared engine limitations without changing SGLang.

The simplified implementation has been tested on H100 with Qwen3-0.6B and unmodified SGLang 993d1fccba, including a regular SMG binary built from that same source. Tests cover aggregated/P/D text, tokens, batches, unary/SSE, logprob options, native errors, disconnect release, existing chat endpoints, and all four P/D DP-rank pairs under KV-aware and round-robin policies. The H100 profiling probe exercises non-generation paths, GET/POST methods and native plain-text replies through the same bridge. These are functional checks, not throughput benchmarks or a complete worker-failure campaign.

Two reference limitations are retained: streamed P/D responses do not receive the prefill prompt-logprob merge, and P/D n=3 timed out through both SMG and Dynamo in the tested stock-engine configuration. Aggregated n=3 succeeds through both. The reference run used DP-aware routing and one attempt to disable retries. No engine fix is proposed.

Implementation

Fork draft PR #76 contains the shared HTTP bridge and SGLang-specific adapter in 1,500 added lines, including wiring and focused regression tests. HTTP forwarding uses reqwest directly; routing lifetime reuses the existing request-lease manager and occupancy reservations. No separate transport wrapper or duplicate cleanup state machine is needed. It also exercises SGLang profiling controls through the shared bridge without generation-specific bookkeeping: a real H100 Nsight capture recorded 16,372 GPU kernels. Profiling is an integration test against a selected sidecar, not a new public frontend route. All implementation branches and PRs remain on the author's fork.

References

• Native unary feature request
• Regular SGLang HTTP P/D router