📝 Originally published (in Japanese) at forge.workstyle.tech.
When building voice-based AI interactions in the browser (avatars, voice bots, streaming AI), you’ll inevitably hit pitfalls stemming from audio physics and browser implementation quirks.
This article compiles 16 traps I encountered during product development, organized in a symptom → cause → solution lookup format.
No need to read from top to bottom—jump straight to the symptom you’re facing.
Echo and Self-Response Issues
1.
Avatar Responds to Its Own Voice (Despite ) Symptom: TTS audio is picked up by the mic, and STT recognizes it as user speech, creating a self-response loop.
Cause: AEC (Acoustic Echo Cancellation) requires a reference signal (the "sound to cancel").
Only the browser's official playback paths ( / WebRTC receiver tracks) serve as references.
Custom playback via Web Audio API does not reliably function as a reference.
Solution: Return TTS audio from the server as a WebRTC remote track and play it via an element.
This eliminates echoes without text-matching workarounds (tested: 99 seconds of continuous speech with speakers on, zero false user turn detections).
2.
Echoes Are Gone, but Speaking Simultaneously with the Avatar Distorts My Voice and Causes Misrecognition Symptom: Only during dual speech, proper nouns get mangled (e.g., "社員数" → "シャインズ"), especially at word beginnings.
Cause: Fundamental AEC trade-off.
To cancel echoes, AEC suppresses/distorts near-end (user) audio during dual speech.
Solution: Mitigate in three layers: ① Increase mic Opus bitrate and enable FEC (see Pitfall 13) ② Provide vocabulary hints to STT (see separate article: use "recent avatar speech" as , not a dictionary) ③ Instruct LLM: "Input is STT transcription with potential errors.
Interpret unnatural words as phonetically similar terms and add confirmation prompts."
3.
Can’t Suppress Audio from Other Apps (Music, Videos) Symptom: Audio/lyrics from a YouTube video opened by the agent keep getting transcribed by STT.
Cause: Browser AEC can only reference audio played by the same tab/app.
Audio from other processes is indistinguishable from human speech to the mic.
Solution: No technical silver bullet.
Combine OS speaker separation (e.g., macOS "Voice Isolation"; request via —ignored on unsupported systems), headphones, and downstream noise rejection (LLM/tool-based filtering). "Only the Beginning Is Unheard" Issues
4.
Initial Speech at Session Start Isn’t Recognized Symptom: First 10–20 seconds of speech go unanswered.
Works normally afterward.
Cause: Two factors: (a) AEC convergence—AEC only learns during actual playback, so unconverged dual speech suppresses the user's voice. (b) ramp-up—Gain increases gradually, making initial speech too quiet.
Solution: (a) Play short greetings/sound effects before the main session to train AEC (loading screens are perfect for this). (b) Set .
If STT is server-side, volume fluctuations are handled well, and disabling AGC has minimal downsides.
Never disable .
5.
Can’t Diagnose "No Response" Issues Symptom: Unclear whether the issue is mic off, preprocessing loss, or server-side—leads to guesswork.
Solution: Add two observability points: ① Server-side logging of input audio energy/probability (10Hz.
Distinguishes "complete silence" from "distorted audio"). ② Level meter in UI for the actual transmitted stream ( to read the same stream as transmission—doesn’t affect transmission).
6.
Starting Mic Track with Causes "No Response" Complaints Symptom: Implementation follows spec ("mic off until button press"), but users expect "always listening." Cause: Not a bug—mismatch between design contract and user experience expectations.
A silent track looks indistinguishable from "quiet room" to the server.
Solution: Align defaults with product promises.
If "always listening" is a selling point, auto-enable on initialization completion (not during connection—that breaks mid-init conversations).
Keep the button as a mute toggle.
Chromium Implementa