[Feature]: Add a provider file-handle arm to InputContentSource

Problem

InputContentSource can express media as inline base64 (data) or as a URL (url), but not as a provider-issued file handle.

Every major provider now has a Files/storage API where you upload an asset once and reference it by an opaque handle on subsequent requests:

These are not interchangeable with the existing two arms:

• An OpenAI or Anthropic file_id is an opaque id, not a URL, so it cannot go in url.
• A Gemini file URI looks like a URL but is auth-gated and only meaningful to Gemini — putting it in url invites any other consumer to try to fetch it.
• Putting either in data is simply wrong; there are no bytes.

The whole point of the handle is that the bytes are not re-sent, so data defeats the feature and url mis-describes it.

Where this bites

In TanStack AI we added a { type: 'file' } content source that carries a per-provider reference record, and adapters map it to each provider's native field. It works fine server-side, but our AG-UI serializer (uiMessagesToWire → RunAgentInput.messages) is the client → server transport for our useChat hook, and a file source cannot cross it. We currently throw a clear error there rather than silently dropping the part or mis-encoding it as a URL. That means the browser can't hold a handle and reference it across turns — the handle has to travel out-of-band in the caller's own payload and be rebuilt server-side, which routes around the protocol.

Any framework wiring a provider Files API through AG-UI will hit the same wall.

Proposal

Add a third arm to InputContentSourceSchema in sdks/typescript/packages/core/src/types.ts (and the peer SDKs), alongside InputContentDataSourceSchema / InputContentUrlSourceSchema. Roughly:

const InputContentFileSourceSchema = z.object({
  type: z.literal("file"),
  // Provider name -> that provider's reference (file_id, file URI, storage URL).
  reference: z.record(z.string(), z.string()),
  mimeType: z.string().optional(),
})

Keying by provider matters: the same bytes uploaded to two providers merge into one source that routes correctly to either, and a consumer reads only its own entry and errors when there is none — rather than guessing whether an opaque string belongs to it. A single { provider, reference } pair would also work if a record is more than the spec wants; the essential part is that the reference is explicitly a provider handle, distinguishable from a fetchable URL.

Since InputContentSource is a discriminated union validated at runtime, this needs a schema change, not just a type change — hence raising it here rather than working around it downstream.

Related

• #126 — the original multimodal-transport proposal (closed); this is the natural follow-on now that providers reference media by handle rather than by value.
• #2133 — tracking spec-compliant InputContent across SDKs; a new arm would need to land there too.

Versions

Checked @ag-ui/[email protected] (latest stable) and 0.1.1-canary.beta.0 — both are data | url.

Happy to help

If the shape is agreeable I'm glad to open a PR for the TypeScript SDK (schema, types, tests) and follow up on the docs.