Support optional mid-session model switching through session.update

Problem

Allow a client to change the configured LLM/model route within an existing Realtime session using session.update, without reconnecting or losing the retained conversation. The selected route may identify a logical model and an explicit provider; deployment-side routing decides which compatible worker serves each request.

This should be an optional extension that preserves existing supported OpenAI Realtime client flows. OpenAI's hosted API does not support changing the session model through session.update, so documentation must distinguish this additional behavior from exact OpenAI API conformance: official session.update reference.

Proposed contract

Reuse the existing event shape, with an illustrative configured route alias:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "model": "reachy-alternative"
  }
}

• Make switching an explicitly supported, documented server capability. Clients that do not request a switch keep their current behavior; no new mandatory event type or client field is required. Route aliases are specific to the configured service, and this operation is not portable to OpenAI's hosted API.
• Initially support switching among configured remote LLM routes handled by the existing adapters. A provider-specific alias can preserve an explicit provider choice without adding a mandatory provider field to the Realtime schema. Do not silently fall back to a different model/provider.
• Accept a switch only at a safe turn boundary. Reject it while a turn/response is pending or in progress, or a tool continuation is unresolved. The client can finish the turn or cancel active generation and wait for cleanup first. Cancellation alone must not make unresolved tool history valid.
• Validate the requested route and the current session's requirements before committing the change, including applicable tools, images, context limits, and continuation semantics. Rejected or failed switches leave the previous effective selection and session configuration intact.
• Apply the selection atomically with respect to new inference admission. A successful session.updated must report the effective new model; subsequent requests must actually use that route. An update that includes other session fields must not leave a partially applied configuration if the switch is rejected.
• Preserve the connection, session/conversation identity, retained history, and compatible session settings. Serialize the retained context for the destination using existing adapters and history policy. Reject switches that require unsupported history conversion; never silently discard required context or carry unsupported backend-local continuation references to another provider.
• Clear or replace the old routing affinity when changing compatible pools. Keep conversation identity stable and propagate it with the selected route on internal inference requests, without requiring public clients to manage worker affinity.
• Return failures through the existing Realtime error mechanism, correlated to the supplied client event ID. Preserve normal response, audio, tool, and cancellation event ordering.

Ownership and integration

speech-to-speech owns the session update, safe switch boundary, conversation serialization, and propagation of logical route/conversation identity through its client adapters. The deployment gateway owns route catalogs, route authorization, provider credentials, compatible worker pools, quotas, scheduling, draining, and autoscaling. Reuse existing configuration and adapters; do not move fleet management into this library.

The deployment revision inspected during design, 3986f453012a131632eee4731995474046846794, records model updates in session configuration, while Chat Completions requests use the handler's model name. Recheck current code when implementing: acknowledging a new session model is not proof that inference switched.

Related work:

• andimarafioti/s2s-endpoint#114: model/provider resolution and independent deployment pools. This issue is its upstream companion; the gateway work remains a separate PR stacked on endpoint PR #110.
• andimarafioti/s2s-endpoint#112: affinity and draining must account for explicit model/pool changes without reusing an incompatible binding.
• andimarafioti/s2s-endpoint#113: continuation contracts and cache verification must distinguish same-pool worker reassignment from an explicit model/provider switch.
• #519: preserve Responses reasoning items and tool-continuation fidelity; reuse that work where needed.
• #484: retain the established official-client compatibility coverage for WebSocket and WebRTC.

Acceptance criteria

• A multi-turn session switches between two configured compatible routes without reconnecting; the next inference request uses the new route and retained context, and session.updated reports it accurately.
• An explicit provider-specific route is respected, and unknown, unauthorized, or incompatible choices fail without changing the effective configuration.
• Tests cover busy/pending turns, unresolved tool calls, cancellation cleanup, concurrent update/admission, rejected combined updates, and disconnect/pipeline reuse. A change in one session cannot affect another session or a reused pipeline slot.
• Tests demonstrate that routing identity remains stable while affinity is isolated by the selected compatible pool; no backend-local response reference or stale work leaks across the switch.
• Existing supported OpenAI client flows remain covered over both WebSocket and WebRTC. Exercise the extension with a pinned official client capable of sending the existing event shape, and document any SDK limitations.
• Documentation explains enabling/using the extension, safe switching and error behavior, supported history/capability boundaries, and its difference from OpenAI-hosted model immutability.

Non-goals

• Implementing deployment pools, fleet lifecycle, shared provider quota policy, or a dynamic provider/plugin framework here.
• Hot-loading arbitrary local model weights or independently changing STT/TTS models through new session fields.
• Migrating an in-flight generation, replaying a partially streamed response, or implementing GPU KV-cache transfer or a distributed response-history store.
• Guaranteeing that every model/provider supports every existing session or all OpenAI Realtime features.