Spec: Authorized Agent Streaming With Gemini Live Rendering

Authorized Agent Streaming With Gemini Live Rendering

Status: ready-for-agent

Source decision: ADR 0005, amendments 2026-07-23

Problem Statement

Voice Mode has two materially different paths. Live · Gemini can reply quickly to casual speech because Gemini Live produces native S2S audio. In contrast, an Agent · Cloud turn currently waits for the entire OpenJarvis agent run to produce one string, then opens a new Gemini Live TTS connection for that complete answer. Manual testing observes roughly 5–8 seconds before agent audio, which does not feel realtime or full-duplex.

This must not be solved by letting Gemini create the authoritative answer for agent-required speech. OpenJarvis remains responsible for Final Voice Turns, memory, permissions, approvals, tools, MCP/local actions, side effects, and persistence. The answer path also needs truthful model provenance: an Agent · Cloud status cannot merely be a label when the configured agent may have used another model.

The system needs low-latency, sentence-level agent audio without releasing a draft, tool result, private fact, approval result, or side effect before the relevant authority work has settled.

Solution

Deepen the existing OpenJarvis Agent Responder into one authorized-response contract with two phases. prepare(Final Voice Turn) performs all required planning, memory reads, tool calls, permission decisions, and approval transitions. It returns either ReadyToAnswer or PendingApproval.

Only ReadyToAnswer may produce Committed Agent Sentence Chunks. Its streaming phase has no access to tool, permission, or side-effect adapters. DeepSeek Cloud is the primary text generator for this phase; local Qwen is available only if the Cloud path fails before the first committed chunk. Gemini Live is an output-only, pre-warmed renderer that receives sequential canonical chunks and returns audio. It receives no tool schema or private agent context.

PendingApproval may produce only one fixed OpenJarvis policy-owned approval prompt. It may not speak LLM-generated rationale, private facts, tool output, or an action promise before confirmation.

The Voice Session owns a cancellation epoch. A new user utterance invalidates the old epoch, stops renderer output, and never resumes it. Renderer fallback to local TTS is permitted only before the first audio byte of the answer. After that played boundary, remaining audio stops while canonical text remains in the UI. Externally started approved actions have an independent durable outcome record; barge-in never means rollback.

The target for a tool-free, agent-required turn is first committed audio P50 at or below 1.5 seconds and P95 at or below 2.5 seconds, measured from the Gemini Live final-turn event. Tool and approval wait time are separately traced.

User Stories

1. As a Vietnamese Voice Mode user, I want a tool-free agent-required question to start speaking promptly, so that protected conversation still feels realtime.
2. As a Voice Mode user, I want casual conversation to remain on Gemini Live native S2S, so that low-risk chat stays natural and fast.
3. As a Voice Mode user, I want sensitive, private, action-oriented, or uncertain speech to remain under OpenJarvis Agent Authority, so that fast native audio never bypasses safety.
4. As a Voice Mode user, I want a fixed acknowledgement while an authorized agent is preparing work, so that a protected turn does not feel stalled.
5. As a Voice Mode user, I do not want that acknowledgement to expose a private fact, tool result, or action promise, so that preparation audio cannot leak authority work.
6. As a Voice Mode user, I want any answer sentence I hear to be committed before it is spoken, so that later tool or approval changes cannot contradict heard audio.
7. As a Voice Mode user, I want a pending approval to use a clear, fixed confirmation request, so that a model cannot persuade or fabricate details before I confirm.
8. As a Voice Mode user, I want the agent response text shown in the UI to be canonical, so that I can rely on the visible answer if speech delivery fails.
9. As a Voice Mode user, I want Gemini speech rendering to follow the committed text without adding meaning, so that rendering does not become another answer authority.
10. As a Voice Mode user, I want speaking over the assistant to stop the old answer immediately, so that my new utterance owns the next turn.
11. As a Voice Mode user, I do not want an interrupted answer to resume automatically, so that I never hear stale context after barge-in.
12. As a Voice Mode user, I want a renderer failure after speech begins to leave the canonical answer visible, so that I can still read the completed information without hearing a duplicated or different voice.
13. As a Voice Mode user, I want a previously approved external action to report only its confirmed outcome, so that interruption is never misrepresented as cancellation.
14. As an OpenJarvis operator, I want Agent · Cloud to identify the actual Cloud model used, so that model provenance is auditable.
15. As an OpenJarvis operator, I want local fallback only before user-visible output starts, so that an answer cannot switch providers halfway through speech.
16. As an OpenJarvis operator, I want a pre-warmed output-only Gemini renderer per Voice Session, so that agent speech avoids a new Live connection cold-start for every answer.
17. As an OpenJarvis maintainer, I want one preparation-to-streaming boundary, so that tools and side effects are structurally unavailable after the first committed audio.
18. As an OpenJarvis maintainer, I want cancellation to invalidate stale chunks and stale persistence, so that async work from an older turn cannot escape into a newer conversation epoch.
19. As an OpenJarvis maintainer, I want tool and approval work to finish before committed streaming starts, so that each audible sentence has a settled authority basis.
20. As an OpenJarvis release owner, I want phase timestamps and actual provider/model IDs, so that first-audio SLO regressions can be attributed to preparation, generation, or rendering.
21. As a privacy-conscious user, I want telemetry to omit raw audio, transcripts, prompts, answer text, and tool arguments, so that latency observability does not become content retention.
22. As a release owner, I want a Vietnamese benchmark that distinguishes casual native latency from tool-free agent first-audio latency, so that a fast casual demo cannot hide a slow protected path.

Implementation Decisions

• Preserve OpenJarvis Voice Agent Authority for identity, Final Voice Turns, memory, persistence, permissions, approval, tool execution, MCP/local action, and durable action outcomes.
• Preserve risk-adaptive Gemini Live native S2S only for clear casual speech. Gemini does not receive OpenJarvis tools and does not become an agent answer authority.
• Evolve the existing Agent Responder contract into a two-phase authorized response seam: preparation returns ReadyToAnswer or PendingApproval; only the ready state exposes committed sentence streaming.
• Make preparation responsible for all required agent planning, memory reads, tool calls, permission checks, approval transitions, and action initiation. Streamed response generation cannot access those adapters and cannot create a new side effect after first audio.
• Permit a fixed, non-authoritative Voice Progress Cue during preparation. It is OpenJarvis-owned, transient, and prohibited from describing private data, tool output, approval outcomes, or intended actions.
• Permit only a fixed policy-owned confirmation prompt for pending approval. LLM-generated explanation starts only after the user confirmation creates a new eligible turn.
• Use DeepSeek Cloud streaming as the primary generator of committed sentence chunks for agent-required answers. Local Qwen is a pre-output fallback only; it may not restart once a chunk is committed or audio begins.
• Treat OpenJarvis committed text as canonical. Gemini Live receives an exact speech-rendering instruction and has no semantic authority. MVP does not pre-buffer audio or wait for output-transcription comparison before playback.
• Replace per-answer Gemini TTS setup with one pre-warmed output-only renderer lifecycle owned by each Voice Session. It accepts sequential committed chunks and receives no tool schema, memory, permissions, agent context, or action capability.
• Keep local rendering fallback only before the agent answer's first audio byte. After the played boundary, renderer failure cancels remaining audio and shows canonical text plus a short recoverable status; it never switches voices, replays, or resumes automatically.
• Give each Voice Turn a session-owned cancellation epoch. A newer utterance invalidates old output immediately and cancels cancellable preparation work. Work that cannot be cancelled may complete internally but cannot emit, persist, or initiate a new side effect for the invalidated turn.
• Store externally initiated user-approved actions in an independent durable outcome record. An interruption never implies rollback; a later turn reports only provider-confirmed status.
• Emit one privacy-minimal Voice Turn Trace: hashed session ID, route and Sentinel verdict/version, phase timestamps, actual provider/model ID, played boundary, outcome/error code, and tool/approval durations. Do not emit PCM, transcripts, prompts, answer text, tool arguments, or account content.
• Treat the existing Gated Realtime Voice Service as the primary orchestration and test seam. It owns observable route, progress, transcript, audio, cancellation, and completion events while delegating authority work to the deep authorized-response contract.
• Keep the existing internal feature flag and allowlist. The Local Voice Baseline remains a fallback path and no public rollout is introduced here.

Testing Decisions

• Test externally visible behavior at the Gated Realtime Voice Service seam, using fake realtime provider, authorized responder, renderer, tool/action adapters, and Voice Session. Avoid asserting private prompt wording or task internals.
• Add contract tests for preparation outcomes: ready turns cannot access tools or permissions while streaming; pending approval emits only the fixed policy prompt; progress cue is fixed and non-authoritative.
• Add streaming tests showing that a tool-free Cloud answer emits a first committed sentence before complete-answer generation finishes, then streams ordered later chunks through the renderer.
• Add authority tests showing that required tool, permission, approval, and memory work settles before the first committed chunk, and that no new tool or side effect begins after first audio.
• Add Cloud-routing tests that expose actual provider/model provenance and prove local fallback is allowed only before first committed output.
• Add renderer lifecycle tests for one pre-warmed output-only renderer per Voice Session, sequential exact input, no tool schema, pre-played-boundary fallback, and no post-boundary switch, replay, or automatic resume.
• Add cancellation-epoch tests for barge-in during preparation and streaming, stale chunks/events, cancellation of cancellable work, and suppression of stale persistence or new side effects from uncancellable work.
• Add durable action tests proving that an already-started action is not called cancelled by an interrupted turn and is reported only after provider-confirmed outcome.
• Add Voice Turn Trace tests proving actual route/model and phase times are emitted while PCM, raw transcript, prompt, answer text, tool arguments, and account content are absent.
• Extend the consented Vietnamese benchmark and release harness with separate tool-free agent first-audio SLO measurements: P50 <= 1.5 seconds and P95 <= 2.5 seconds from Gemini Live final-turn event. Preserve existing risk-safety, native-casual latency, and unintended-downgrade gates.
• Keep existing Voice Session, Gated Realtime Voice Service, realtime TTS, approval, playback-boundary, sentinel, and browser VAD tests as regression coverage.

Out of Scope

• Changing the Gemini Live casual-native risk-adaptive policy or Sentinel classifier design already covered by ADR 0005.
• Giving Gemini tools, memory, permissions, persistence, or agent-answer authority.
• A second STT pipeline or pre-playback output-transcription verification.
• Mid-answer local/cloud rerouting, voice switching, replay, or automatic resume after a renderer failure or barge-in.
• General redesign of text-chat agent tools, MCP permissions, or durable action storage beyond the Voice Turn integration boundary.
• Public enablement, removal of the Local Voice Baseline, or retention of raw conversation content for observability.

Further Notes

• The primary test seam is intentionally the existing Gated Realtime Voice Service plus the Agent Responder contract. It is the highest existing seam that observes Final Voice Turns, agent handoff, renderer playback, route, cancellation, and completion without creating a second orchestration layer.
• The existing responder currently waits for a whole answer before emitting text, and the current Gemini renderer opens a connection per answer. This spec replaces those latency-producing contracts rather than treating the observed 5–8 second agent path as a VAD or Sentinel bug.
• The first-audio SLO excludes required tool and approval wait, but their trace durations remain visible. This prevents a metric from hiding authority cost.
• This is an internal High MVP behind the existing allowlist and feature flag. Manual browser testing is useful but does not replace contract, benchmark, or privacy tests.