[RFC]: Request level text and derender output on `/inference/v1/generate`

Motivation.

The disaggregated serving path in vLLM is render → generate → derender. /render turns an OpenAI request into a GenerateRequest (#36166). /inference/v1/generate runs it on the GPU tier and returns token IDs (#22817, #24261). /v1/chat/completions/derender and /v1/completions/derender turn the result back into an OpenAI response (#42729). This works well for batch and RL traffic where the caller already holds the complete GenerateResponse.

Latency sensitive interactive traffic in prefill/decode (P/D) deployments is a poor fit. Streaming derender (#47161) is a stateless per chunk translator, so every SSE chunk from generate costs the client another HTTP round trip:

• One POST per chunk with the client echoing stream_state back on each call .
• On the parsing path the full chat_request is re-sent with every chunk.
• Parser state is rebuilt by replaying all output tokens each chunk. #50550 documents this as O(n³) character work over a generation and defers caching it.

All of this lands on the TPOT critical path and for parsed output the cost grows with response length. A benchmark of the streaming derender PRs measured it (https://github.com/vllm-project/vllm/issues/56851#issuecomment-5685698033): 1×A100-40G, Qwen3-8B, hermes + qwen3 parsers, main plus #50550). At matched load, detokenize and parse on the derender tier cost ~9× the in-process CPU, and E2E p50 rose 15% on multi-thousand-token reasoning outputs. Tokens-in / text-out in a single call already works on the main server (/v1/completions accepts token IDs in prompt). The gap is specific to deployments whose wire format is GenerateRequest.

Tokenizer availability is per pool. Detokenization is already per request. Only one of these is a launch time decision:

	Granularity	Where it's decided today
Tokenizer loaded	per pool, at launch	--tokens-only sets skip_tokenizer_init
Detokenize this request	per request	SamplingParams.detokenize, default True. GenerateRequest.sampling_params is a full SamplingParams, so it's on the wire today.

The only thing that makes detokenization look launch level is --tokens-only overriding the request value. A generate server launched with --enable-scale-out and without --tokens-only already runs a FastIncrementalDetokenizer for every request. The handler then builds the response from output.token_ids only and throws output.text away.

P/D makes request level the natural grain:

• Only the terminal leg's output reaches the client. Proxies run the prefill leg with max_tokens=1 and keep only kv_transfer_params. Decode recomputes the last prompt token to sample the first output token itself.
• Decode already has everything detokenization needs. GenerateRequest.token_ids carries the full prompt and the detokenizer primes its DecodeStream from it. No detokenizer state crosses the P→D boundary.
• State lives with the live request. RequestState owns the detokenizer and is popped on finish and on abort/disconnect. There's no session registry or eviction. That's the property #47161 couldn't get on the stateless derender tier which has no live request to hang state on.
• No new pool flavor. P/D fleets already run prefill and decode pools with different launch configs. Prefill pools stay --tokens-only. Decode pools load a tokenizer.

This also continues #22817's original design, which says "We can make Detokenization step optional in the AsyncLLM" and sketches a service mapping GenerateRequest → ChatCompletionResponse.

Proposed Change.

1. A request-level output_mode field

Add a top level field to GenerateRequest:

output_mode: Literal["tokens", "text", "derender"] = "tokens"

It's a new field rather than a reuse of sampling_params.detokenize. That field defaults to True, so keying response content off it would change every existing response on tokenizer loaded servers. It's also an engine field and response shape is a serving concern.

Level	Response adds	Equivalent today	Server needs
tokens (default)	nothing, unchanged behavior	/inference/v1/generate	nothing
text	detokenized text, resolved logprob tokens	/v1/completions/derender	tokenizer
derender	content, reasoning, tool_calls	/v1/chat/completions/derender	tokenizer, parser config, per request parser context

2. Response shape (proposed, see open question 1)

Keep the generate format and add optional fields that reuse existing OpenAI types:

class GenerateResponseChoice(BaseModel):
    ...  # index, logprobs, finish_reason, token_ids, routed_experts, sampling_mask
    text: str | None = None             # output_mode="text"
    message: ChatMessage | None = None  # output_mode="derender"

class GenerateResponseStreamChoice(BaseModel):
    ...
    text: str | None = None             # output_mode="text", delta text
    delta: DeltaMessage | None = None   # output_mode="derender"

• One response schema for the endpoint regardless of mode. Generate only fields (token_ids, routed_experts, sampling_mask, kv_transfer_params) keep their place.
• message / delta are the same types /derender and /v1/chat/completions emit, so the parity check compares field for field.
• New fields are None and omitted unless requested, so existing clients see no change.

Example decode-leg request and stream chunk:

{"token_ids": [151644, 872, ...], "sampling_params": {"max_tokens": 512},
 "stream": true, "output_mode": "text", "kv_transfer_params": {...}}

{"request_id": "...", "choices": [{"index": 0, "token_ids": [9707], "text": " Hello"}]}

3. Parser context for the derender level (proposed, see open question 2)

Parsers are built per request from context GenerateRequest doesn't carry today: tools, tool_choice, chat_template_kwargs. Batch /derender solves this by taking the post adjust_request ChatCompletionRequest. The proposal mirrors it:

chat_request: ChatCompletionRequest | None = None  # used only when output_mode="derender"

• It's sent once per request, not once per chunk as with streaming derender.
• It reuses the batch derender path (OnlineDerenderer.derender_chat) and its parity tests with stop string handling added (section 9). That path runs on the renderer's executor which has renderer_num_workers threads, default 1. It runs once per request and decode pools do no prompt tokenization, so contention is low but the pool size caps concurrent derender calls.
• Without chat_request, a server with a reasoning or tool parser configured returns 400 (section 6). Batch /derender falls back to plain detokenization instead which can mix reasoning or tool call markup into message.content. Streaming derender already fails closed for parser configured models. Clients that want plain text use text. On a server with no parser, derender without chat_request puts the plain text in message.content.

4. Streaming

• text: text carries each chunk's delta from the request's existing incremental detokenizer (RequestOutputKind.DELTA). No extra state.
• derender: one Parser per choice, created when the request starts and kept for its lifetime, the same way /v1/chat/completions streams today. There's no replay, no stream_state and no per chunk chat_request.
• End of stream: at text and derender levels, the stream emits a chunk whenever the engine output carries new text or a finish_reason, even with no new token IDs. That covers the final output the engine produces on /abort_requests which has no token IDs but can flush text held back for stop string matching. The generate stream skips outputs with no new token IDs today. On main, #53187 only keeps them while prompt metadata is pending), so tokens streams already drop a finish only chunk. Changing that for tokens alters the existing stream, so it's a separate fix.

5. Logprobs

At text and derender levels, logprob entries carry decoded token strings and bytes, as /derender produces, instead of the token_id:N placeholders generate emits today. At the tokens level nothing changes.

Dense logprobs are the expensive case. At top_logprobs=20 the benchmark measured 355 µs per logprob-bearing token and 0.83 of a frontend core within 1.2× of the GPU's token rate. On a tokenizer loaded server that cost is already paid at the tokens level: LogprobsProcessor decodes every top-k token whenever sampling_params.detokenize is trueand generate then discards the strings. Clients that want wide logprobs off the decode host sendoutput_mode="tokens"withdetokenize: false(which rules outstopstrings) or use a--tokens-only` pool. The Phase 1 docs should say so.

6. Validation (fail loud)

• output_mode != "tokens" on a server without a tokenizer → 400. Today's no-op detokenizer returns "", so a naive implementation would return empty text with a 200 for any request routed to a --tokens-only pool.
• output_mode != "tokens" with sampling_params.detokenize explicitly false → 400.
• chat_request set without output_mode="derender" → 400, rather than silently ignoring it.
• output_mode="derender" without chat_request on a server with a reasoning or tool parser configured → 400, rather than falling back to plain text that may contain reasoning or tool call markup.

Rollout note: GenerateRequest is a plain pydantic BaseModel, so servers without this change ignore output_mode and return tokens only with a 200. Clients should check for text / message in the response.

7. Role agnostic P/D usage

The field has no behavior tied to pod role or kv_transfer_params. The same decode pod may serve some requests decode only and others after a remote prefill, so role keyed behavior would differ across those paths. Orchestrators set output_mode and chat_request on the terminal leg only. If they reach a tokenizer free prefill pool anyway, the 400 above surfaces the misconfiguration.

For llm-d: the coordinator builds the prefill request from scratch, so it needs no change. The NIXL v2 sidecar reuses the client body for prefill and overrides a fixed set of fields (stream, token limits, kv_transfer_params). It would need to reset output_mode and drop chat_request on the prefill request, keeping both for decode. Resetting only output_mode leaves chat_request on a tokens request, which section 6 rejects.

8. Decoupling --tokens-only

A decode pool needs the generate API surface with a tokenizer. --tokens-only currently does four jobs:

Job	Where	Proposal
Skip tokenizer init	[arg_utils][tokens-only-tokenizer]	keep
Force detokenize=False	[factories][factories-tokens-only] → [serving][force-no-detok]	keep
Register /abort_requests	[api_router][abort-gate]	also register when generate is served on a tokenizer loaded server (open question 4)
Imply scale-out endpoints	factories.py:73	keep

A decode pool then runs --enable-scale-out with a tokenizer and without --tokens-only.

9. Parity and shared code

The inline path and /derender must produce identical output or clients get different text depending on deployment.

• Both go through the same Parser code, with no parallel copy.
• Extend the round-trip parity test (#48617) to assert generate(output_mode=X) matches derender(generate(output_mode="tokens")) for text and derender, batch and streaming.
• Streaming derender drops logprobs on main, so the streaming logprob checks depend on #55029.
• Stop strings. Inline levels follow the engine's stop handling, like /v1/completions and /v1/chat/completions: a matched stop string is cut from the text unless include_stop_str_in_output is set. Token IDs keep every generated token, so decoding them in /derender brings the stop string back, plus anything after it in the same token. For requests with stop, the test asserts inline output matches the coupled endpoint and that /derender output starts with the inline text and differs only after it. The tokens leg has to run on a tokenizer-loaded server, since --tokens-only ignores stop today (related bug below). OnlineDerenderer.derender_chat` re-decodes token IDs, so the Phase 2 inline path has to apply the stop truncation before parsing.
• End of stream. Tests cover a final chunk carrying only buffered text or only a finish_reason, including after /abort_requests (section 4).

10. Rust frontend

rust/src/server/src/routes/inference/generate/types.rs mirrors GenerateRequest / GenerateResponse. The protocol change should be agreed here once and mirrored there. The Rust derender level depends on the Rust /derender work (#53223).

11. Phasing

Phase	Scope
1	output_mode field, text level (batch and streaming), logprob resolution, validation, /abort_requests decoupling, docs, latency benchmarks (section 12)
2	derender level, batch: chat_request context, message on the choice, reusing OnlineDerenderer.derender_chat
3	derender level, streaming: request-scoped parsers, delta on stream choices, parity CI extension, latency benchmarks (section 12)
—	Rust frontend mirror, tracked with the Rust frontend owners

12. Latency benchmarks

The CPU numbers above don't cover latency. The Phase 1 and Phase 3 PRs report TTFT and TPOT for:

• inline text / derender against generate plus streaming derender, across concurrency levels and response lengths.
• tokens requests on the same pod, with and without text or derender traffic mixed in.

Alternatives considered

1. Streaming derender as-is (#47161). Keeps the GPU tier tokenizer free but puts a round trip and echoed state on every chunk, plus replay for parsed output. It stays the right tool for stateless postprocessing (RL, batch, gateways that own postprocessing). This RFC complements it and doesn't replace it.
2. The gateway composes generate + derender. The per chunk hop moves from client to gateway rather than going away. Holding state in the gateway instead means reimplementing postprocessing there.
3. Server-side derender sessions keyed by request_id. Rejected in #47161 (Option B) over session lifecycle, eviction and memory. The derender tier has no live request to attach state to. The generate server does.
4. Token IDs on /v1/chat/completions. Chat has no token ID input (messages is required). Even with one, a render produced GenerateRequest carries resolved SamplingParams that don't map back onto OpenAI request fields without re-resolving and orchestrators on the generate format would have to run two wire formats.
5. Run the coupled OpenAI server on decode pools. Decode pods leave the generate protocol with the same two format problem as (4).
6. Reuse sampling_params.detokenize as the switch. It defaults to True, so every existing response on a tokenizer loaded server would grow text and token only clients would pay for detokenization by default.

Costs and risks

• CPU on the decode host. Detokenization and parsing run in the API server process, not the engine core. They don't block the scheduler but they use host CPU. text adds little on a tokenizer loaded server because detokenization already runs there today. Parsing and logprob resolution are the new work. In the benchmark, with the GPU saturated, the in-process frontend stayed under 0.25 of a core for chat, reasoning and tool-calling traffic (24–144 µs per token), at least 3.5× headroom over the GPU's token rate. That was one A100 and a faster GPU narrows the gap. --api-server-count scales the frontend if it becomes the limit. Dense logprobs are the exception (section 5). Opt-in per request bounds the cost to the traffic that asks for it.
• Parser config drift. Decode pools need the same --reasoning-parser, --tool-call-parser and --enable-auto-tool-choice as the render tier. That's the same obligation the derender tier already has.
• Two call sites for postprocessing. Mitigated by shared Parser code and the parity CI in section 9.
• Security. The inline path postprocesses engine produced tokens bounded by the request's resolved limits. It doesn't accept caller supplied output token IDs the way /derender does, so it doesn't need /derender's explicit bounds checks. chat_request is caller supplied and needs the same validation as on the chat and derender endpoints.

Open questions

1. Response shape. An extended generate format with text / message / delta (proposed) or return ChatCompletionResponse / ChatCompletionStreamResponse directly for output_mode="derender"? The latter is a drop-in for existing /derender consumers but the endpoint's response schema would then depend on a request field and the generate only fields would need a home.
   • Feedback so far: @nilig supports extending the generate response format (https://github.com/vllm-project/vllm/issues/56851#issuecomment-5692634586).
2. Parser context. Embed the post-adjust_request chat_request (proposed), or have /render emit a slim derender context (tools, tool_choice, chat_template_kwargs, plus adjust_request-derived state) that clients pass through? The slim form is smaller but first needs a settled answer to the open adjust_request question from #42729.
3. Naming. output_mode with tokens | text | derender, or something else?
4. /abort_requests. Register it whenever generate is served, add a separate flag or leave it tied to --tokens-only?
   • Feedback so far: @nilig wants it whenever generate is served, including on decode pools that load a tokenizer (https://github.com/vllm-project/vllm/issues/56851#issuecomment-5692634586).
5. Completions shaped parsing. Is text enough for completions shaped clients or does anyone need reasoning/tool parsing on completions? Parsing on /v1/completions/derender is chat-only today.
   • Feedback so far: @shimib's logprob-heavy traffic is completions shaped and needs text plus resolved logprobs, not parsing (https://github.com/vllm-project/vllm/issues/56851#issuecomment-5685698033).

Feedback Period.

No response

CC List.

@chaunceyjiang @DarkLight1337 @vMaroon @NickLucche @nilig @njhill @robertgshaw2-redhat @sagearc

Any Other Things.

Related work

• #47161 / #50550: streaming derender and its parse path. Complementary to this RFC (Alternatives, item 1).
• #47313: adds stop_reason to generate response choices. It touches the same protocol and serving files. and the matched stop string would let /derender cut its text there too (section 9).
• #36983 / #51641: delta_text and delta_token_ids desync when stop strings are buffered. Streaming text inherits this and should use the fix.
• #53223: Rust /derender endpoints.
• #55029: resolved logprobs on streaming derender chunks. The streaming logprob parity checks in section 9 need it.

Related bug, filed separately: #TBD. On --tokens-only servers, stop strings are silently ignored. SamplingParams validates while detokenize is still True, then the server sets it to False and the no-op detokenizer never matches. Generation runs to EOS or max_tokens with a 200. It should be a 400, independent of this RFC. Once decode pools run with a tokenizer, stop also works there.

Out of scope

• Skipping detokenization for output_mode="tokens" on tokenizer loaded servers where today it runs and its output is discarded. It's a worthwhile performance follow up but it changes existing behavior (stop strings need detok).
• Any change to /derender itself.

Before submitting a new issue...

• Make s