[FEATURE]: Give vLLM-Omni OutputFormatter a typed modality contract
Feature request
Refactor the vLLM-Omni OutputFormatter boundary so callers know the request-level formatting contract without knowing which concrete formatter will be selected.
This became more visible while adding joint video/audio handling in #13707. OutputFormatter is shared by the aggregated handler and disaggregated stage router, but it currently centralizes dispatch without fully abstracting the concrete formatter calling conventions.
Problem
- Dispatch has two authorities:
stage_output.final_output_typeselects a formatter, whilerequest_typeseparately distinguishes image and video insideDiffusionFormatter. OutputFormatter.format(..., **ctx)is an untyped union of concrete formatter arguments. Callers must know that text consumesprevious_text, video consumesfpsand format options, and audio consumes speed plus stream/aggregate state.- Unknown or modality-inappropriate context keys are generally ignored.
OmniHandlerpasses the full union on every stage, whileOmniStageRouterreconstructs a different subset. Nonemeans several different things: unsupported modality, missing output, an expected streaming gap, or successful audio buffering. Callers interpret it differently; the stage router treats it as an error.- The concrete formatters do not share one interface: text formatting is synchronous and receives
request_output, media formatting is asynchronous and receivesstage_output, and aggregate audio requires a separatefinish_audio()lifecycle. - Failure behavior is mixed between raised validation errors and serialized
status="failed"responses. - The abstraction also owns tensor normalization, encoding, audio/video muxing, storage, and protocol serialization, so the name and contract understate its responsibilities.
The caller should need to know the request modality and requested response behavior, but not that the request routes through a particular concrete formatter with private context keys.
Proposed direction
- Normalize
final_output_typeandrequest_typeonce into one authoritative modality before dispatch. - Replace
**ctxwith a typed request-levelFormatContext(or equivalent explicit typed parameters), validated at theOutputFormatterboundary. - Give modality adapters a consistent asynchronous interface accepting the normalized stage and context.
- Replace overloaded
Nonewith an explicit outcome that distinguishes emitted output, an expected skip/buffer operation, and failure. - Model aggregate-audio finalization explicitly, either through a common
finish()operation or a request-local formatting session. - Validate construction requirements such as writable media storage when URL output is requested.
- Keep encoding and storage implementation details behind the adapter boundary; consider renaming the abstraction if it continues to own the full output pipeline.
This can be implemented incrementally; it should not require changing the public HTTP response schemas.
Acceptance criteria
- Aggregated and disaggregated callers use the same typed formatting contract.
- Callers do not need to know which concrete formatter is selected.
- There is one authoritative modality-resolution step.
- Misspelled or invalid context fields are detected rather than silently ignored.
- Expected no-emission states are distinguishable from unsupported or failed formatting.
- Audio streaming and aggregation state ownership/finalization are explicit in the interface.
- Existing text, image, video, and audio behavior remains covered by focused tests.
Related work
- #13707 adds joint video/audio response handling and exposes the growing modality-specific context.
- #13805 proposes moving final media persistence from the disaggregated router to the final worker. That placement decision should use the same typed formatting contract rather than duplicate formatter-specific knowledge.
Priority and effort
Medium priority: the current implementation works, but the weak boundary raises regression risk as modalities and consumers are added. Expected effort is medium-to-large because both aggregated and disaggregated callers must migrate together.
Source: ai-dynamo/dynamo