[Feature]: Symmetric multimodal output with typed `OutputContent[]` on `AssistantMessage`

Pre-flight Checklist

• I have searched existing issues and this hasn't been requested yet.

Problem or Motivation

AG-UI has formalized multimodal input: UserMessage.content accepts string | InputContent[], with typed Text / Image / Audio / Video / Document parts and data / url sources.

The output side never followed. AssistantMessage.content is still string only. There is no typed way for an agent to send media (an image, an audio clip, a generated file) back to the user. The asymmetry is real and currently undefended: it was raised for both user and assistant messages in #126, but only the input half shipped, and the "what about messages from the agent?" question was asked twice in that thread and never answered before it was closed. The same need keeps resurfacing in discussions (#527 "Support for Images & Files", #168 "How to send ImageMessage", #526 "artifacts in A2A").

Today, agent-produced media is only possible through app-specific conventions: tool calls + generative UI, shared state, or CUSTOM/RAW events. Those are fine for a React frontend, but they leave 4 gaps that a protocol is supposed to close:

1. Fragmentation. Without a typed shape, every framework and app invents its own output-media convention. This is the exact concern that motivated #126 in the first place and it now applies to the half of the conversation that was left out.
2. Non-UI consumers. A2A agent-to-agent flows, CLIs, mobile clients, and eval/logging pipelines have no frontend to render a tool call into an <img> or audio player. They need the media typed in the message to consume it at all.
3. Native multimodal-output models. Models that emit images/audio as first-class output (e.g. Gemini) have no clean channel. You must wrap genuine model output in a synthetic tool call.
4. Round-trip fidelity. Media that isn't part of the message model can't be faithfully represented in MESSAGES_SNAPSHOT, persisted, or replayed. (Media metadata is already lossy even on the input side; see #1809 and #2011 which suggests the message model under-specifies media generally.)

The result is that AG-UI, a protocol whose whole purpose is connecting agents to users, can carry rich media to the agent but not from it.

Proposed Solution

Make output symmetric with input by allowing typed content parts on assistant messages:

interface AssistantMessage {
  id: string
  role: "assistant"
  content?: string | OutputContent[]   // was: string
  toolCalls?: ToolCall[]
  name?: string
  encryptedContent?: string
}

type OutputContent =
  | TextOutputContent
  | ImageOutputContent
  | AudioOutputContent
  | VideoOutputContent
  | DocumentOutputContent

Each non-text part carries a source and optional metadata, reusing the existing input source types rather than inventing new ones (ideally promoted to a single shared ContentSource):

// shared with the input side
type ContentSource =
  | { type: "data"; value: string; mimeType: string }  // inline / base64
  | { type: "url"; value: string; mimeType?: string }  // reference

Backward compatibility is preserved exactly as it was on the user side: content stays string | OutputContent[], so existing text-only agents and clients are unaffected.

Open design question: how it streams. This is the main thing worth deciding together. MESSAGES_SNAPSHOT can already carry the assembled message, but live streaming today is text-only (TEXT_MESSAGE_*). A pragmatic v1 could deliver each media part as a discrete, complete part (URL-first to avoid large base64 over the stream) via a single new content-part event, deferring true inline/streamed binary to a later iteration. I'd rather align on the approach than prescribe one.

Happy to hear from you, learn and adjust.

Alternatives Considered

• Tool calls + generative UI (status quo). Great for rich React frontends and the right tool for interactive output. But it's an app-level convention, not portable across frameworks, and invisible to any non-UI consumer.
• Shared state (STATE_SNAPSHOT / STATE_DELTA). Good for persistent artifact panels, but it's a side channel, the media isn't part of the turn, so association and ordering with the message are implicit.
• CUSTOM / RAW events. The escape hatch that works today and precisely the fragmentation #126 warned about.
• A2A artifacts (#526). Solves a different layer; doesn't address the AG-UI message model itself.
• Markdown / data: URIs inside the content string. Fragile, untyped, no metadata, and unworkable for audio/video.

These each have a legitimate place; none of them gives the protocol a standard, interoperable representation of agent-produced media, which is what this proposal is about.

Additional Context

No response