Listener backchannels abort active response mid-playback
While testing the browser realtime demo, I noticed that uttering natural listener acknowledgements/backchannels (such as "mm-hmm", "uh-huh", "alright", or "go ahead") while the assistant is speaking immediately aborts the in-progress response, clears the audio playback, and triggers a brand-new reply answering the acknowledgement.
I might be missing an existing setting or flag for this, but I wanted to check if this is intended behavior or if there is interest in allowing short passive backchannels to pause and resume playback instead of triggering a destructive cancel.
Steps to Reproduce
- Start the server:
speech-to-speech serve. - Connect via the browser demo at
http://localhost:8765(or connect viatalk). - Ask a question that produces a long response (say "Explain how the solar system formed").
- While the assistant is actively speaking sentence 1, say "mm-hmm" or "right".
Observed Behavior
- The assistant abruptly cuts off mid-sentence.
- The queued audio for the rest of the explanation is cleared.
- The model receives "mm-hmm" as a new user prompt and begins answering it from scratch.
Expected Behavior
Brief listener feedback/backchannels should ideally not abort the active assistant turn. Instead, playback could briefly pause, and then resume the existing response once the short vocalization ends (saving full cancellations for substantive interruptions or queries).
Environment
- Version:
speech-to-speechv1.0.0 (mainat16d7f98) - Platform: macOS (Apple Silicon) / Linux
Screenshot
Source: huggingface/speech-to-speech