#14907·dynamo

[FEATURE]: Give vLLM-Omni OutputFormatter a typed modality contract

Author: GuanLuoCreated Sep 16, 2026Updated Sep 19, 2026
Labelsenhancementlanguage::pythondynamo-llmbackend::vllmrefactormultimodal

Feature request

Refactor the vLLM-Omni OutputFormatter boundary so callers know the request-level formatting contract without knowing which concrete formatter will be selected.

This became more visible while adding joint video/audio handling in #13707. OutputFormatter is shared by the aggregated handler and disaggregated stage router, but it currently centralizes dispatch without fully abstracting the concrete formatter calling conventions.

Problem

  • Dispatch has two authorities: stage_output.final_output_type selects a formatter, while request_type separately distinguishes image and video inside DiffusionFormatter.
  • OutputFormatter.format(..., **ctx) is an untyped union of concrete formatter arguments. Callers must know that text consumes previous_text, video consumes fps and format options, and audio consumes speed plus stream/aggregate state.
  • Unknown or modality-inappropriate context keys are generally ignored. OmniHandler passes the full union on every stage, while OmniStageRouter reconstructs a different subset.
  • None means several different things: unsupported modality, missing output, an expected streaming gap, or successful audio buffering. Callers interpret it differently; the stage router treats it as an error.
  • The concrete formatters do not share one interface: text formatting is synchronous and receives request_output, media formatting is asynchronous and receives stage_output, and aggregate audio requires a separate finish_audio() lifecycle.
  • Failure behavior is mixed between raised validation errors and serialized status="failed" responses.
  • The abstraction also owns tensor normalization, encoding, audio/video muxing, storage, and protocol serialization, so the name and contract understate its responsibilities.

The caller should need to know the request modality and requested response behavior, but not that the request routes through a particular concrete formatter with private context keys.

Proposed direction

  1. Normalize final_output_type and request_type once into one authoritative modality before dispatch.
  2. Replace **ctx with a typed request-level FormatContext (or equivalent explicit typed parameters), validated at the OutputFormatter boundary.
  3. Give modality adapters a consistent asynchronous interface accepting the normalized stage and context.
  4. Replace overloaded None with an explicit outcome that distinguishes emitted output, an expected skip/buffer operation, and failure.
  5. Model aggregate-audio finalization explicitly, either through a common finish() operation or a request-local formatting session.
  6. Validate construction requirements such as writable media storage when URL output is requested.
  7. Keep encoding and storage implementation details behind the adapter boundary; consider renaming the abstraction if it continues to own the full output pipeline.

This can be implemented incrementally; it should not require changing the public HTTP response schemas.

Acceptance criteria

  • Aggregated and disaggregated callers use the same typed formatting contract.
  • Callers do not need to know which concrete formatter is selected.
  • There is one authoritative modality-resolution step.
  • Misspelled or invalid context fields are detected rather than silently ignored.
  • Expected no-emission states are distinguishable from unsupported or failed formatting.
  • Audio streaming and aggregation state ownership/finalization are explicit in the interface.
  • Existing text, image, video, and audio behavior remains covered by focused tests.

Related work

  • #13707 adds joint video/audio response handling and exposes the growing modality-specific context.
  • #13805 proposes moving final media persistence from the disaggregated router to the final worker. That placement decision should use the same typed formatting contract rather than duplicate formatter-specific knowledge.

Priority and effort

Medium priority: the current implementation works, but the weak boundary raises regression risk as modalities and consumers are added. Expected effort is medium-to-large because both aggregated and disaggregated callers must migrate together.