Reduce barge-in latency with reversible pre-confirmation audio ducking
Problem
Realtime sessions can already receive microphone audio while the assistant is speaking, but audible interruption waits for the confirmed SpeechStartedEvent. With the default min_speech_ms=384, the demo may continue playing the assistant for roughly four tenths of a second after the user begins a barge-in.
Lowering min_speech_ms globally is not a safe fix because short noise bursts would become real conversation turns and cancel model work.
This is a focused follow-up to #134. It should remain compatible with the speculative turn lifecycle in #308.
Proposed design
Introduce a reversible pre-confirmation phase:
- VAD emits an internal speech-candidate event after a short configurable amount of active speech.
- WebSocket and WebRTC demo clients reduce output gain immediately, but do not cancel the response or mutate conversation state.
- If VAD confirms the normal
speech_startedturn, the existing path performs the hard cancel and flush. - If the candidate ends below
min_speech_ms, the client restores output gain and the response continues.
The candidate events are an optional protocol extension. Clients that do not understand them continue to use the existing confirmed interruption behavior.
Acceptance criteria
- Candidate detection never creates a conversation item, increments usage, or cancels LLM/TTS work.
- A false candidate restores normal output without losing buffered audio.
- A confirmed candidate still uses the existing cancellation and generation-discard path.
- WebSocket and WebRTC demo clients apply the same gain behavior.
- Candidate state resets on session reset and disconnect.
- Unit and integration tests cover candidate start, false-candidate release, and confirmed transition.
- Existing interruption tests remain green.
Out of scope
- Feeding partial transcripts to the LLM.
- Speculative tool execution.
- Replacing STT, LLM, or TTS models.
- Changing the speculative turn ownership refactor in #308.
Source: huggingface/speech-to-speech