[FEATURE_REQUEST] Gapless playback for streaming TTS providers (mechanism already exists in-tree)

Have you searched for similar requests?

Yes

Is your feature request related to a problem? If so, please describe.

Streaming TTS providers play audio with an audible silence gap (~0.3-0.5s) between consecutive chunks, which sounds like the speech is being cut mid-sentence. With several chunks it becomes genuinely hard to listen to.

This is NOT a network or provider problem. I measured a streaming TTS provider that produces audio 4.19x faster than realtime:

Text                122 chars
Audio produced      24.80 s   (24 kHz PCM16)
Time to produce     5.92 s    (first byte at 1.57 s)
Producer / playback ratio: 4.19x

t=2.09s -> 1.76s of audio received  (surplus +1.24s)
t=4.01s -> 12.96s of audio received (surplus +10.52s)
t=5.91s -> 24.80s of audio received (surplus +20.46s)

Because the producer is much faster than playback, after ~1.7s the client permanently holds more audio than it needs. The buffer never runs dry, yet the audible gaps remain. So the gaps are introduced entirely by the player.

Describe the solution you'd like

The gapless playback mechanism already exists in this repository - it is just scoped to a single provider. tts-webui.js implements it via initAudioWorklet() + processStreamingAudio(), using an AudioContext and the shared lib/pcm-processor.js AudioWorklet, and feeds PCM chunks straight to the sink instead of swapping an <audio> source. PR #5732 reimplements the same approach inside chatterbox.js.

Proposal: extract that PCM sink into the core TTS player so any streaming provider can opt into it, and remove the per-provider copies.

Concretely, in public/scripts/extensions/tts/index.js:

• Route streamed chunks through a single AudioContext + AudioWorklet sink (reusing the existing lib/pcm-processor.js) instead of assigning audioElement.src for every chunk in playAudioData().
• Feature-detect AudioWorklet (it requires a secure context) and keep the current per-chunk path as a fallback.
• Providers that already emit raw PCM/WAV then need no changes, and duplicated per-provider streaming implementations can be deleted.

This directly answers the concern raised when a per-provider version of this was proposed before (PR #5302, closed): the mechanism belongs to the player, not to one vendor.

Why the current player cannot be fixed on the provider side:

• The player is strictly sequential. The next chunk is only taken from the queue after the current one fires ended, so prefetching cannot even start early.
• Every .src assignment forces a new decode and a canplay wait (~0.3-0.5s on mobile), so N chunks always produce N-1 unavoidable gaps.
• Larger chunks only reduce the number of gaps; they can never reach zero.

Describe alternatives you've considered

• Preloading / prefetching the next chunk: does not help. The player is strictly sequential (the next chunk is only taken from the queue after the current one fires ended), so the next download cannot even start early. And even with the bytes in hand, assigning .src still requires a fresh decode + canplay wait.
• Passing a pre-built data-URL string instead of a Blob: this skips the getBase64Async() conversion step (the framework accepts a string directly), so it shaves part of the cost, but it cannot remove the source-swap decode, so gaps remain.
• Larger chunks: fewer gaps, but proportionally higher time-to-first-audio. It only reduces the number of gaps; it can never reach zero.
• Crossfading the chunk boundaries: not applicable. Chunks are concatenated, not overlapping - there is no overlapping material to crossfade. The gap is silence in the timeline, not missing samples, so crossfading would either repeat or drop ~0.3s of actual speech.
• MediaSource: would also work, but reusing the in-tree AudioWorklet + lib/pcm-processor.js has the advantage of already existing and already having been accepted once (tts-webui.js), so it is less new surface for maintainers to review.

Trade-off I accept today: "fast first audio" and "no gaps" are mutually exclusive with the current player, so providers must expose both modes and let the user choose.

Additional context

Environment: SillyTavern 1.18.0, release, commit 8172dcd (2026-07-07) - verified this is still the state of the release branch. Checked the current tts/index.js on release: MediaSource / appendBuffer / AudioContext: 0 occurrences; new Audio(: 1; no provider stop() callback yet.

Note that the gapless sink is not new work: lib/pcm-processor.js already exists in-tree and is already used from tts-webui.js, so this is a matter of extracting and sharing it rather than introducing a new mechanism.

Measurement method (Node 22, streaming endpoint, counting decoded chunk bytes against wall-clock time):

const t0 = Date.now();
const res = await fetch(ENDPOINT, { method:'POST', headers:{...},
    body: JSON.stringify({...stream:true}) });
const reader = res.body.getReader();
// per SSE frame: bytes += atob(frame.choices[0].delta.audio.data).length
// ratio = (bytes_total / 48000) / ((Date.now() - t0) / 1000)

Reproduction case: I hit this while writing a third-party provider for Xiaomi MiMo (https://github.com/ymf2317-tech/SillyTavern-MiMoTTS). MiMo is a good stress case because it is slow to finish a whole utterance but streams fast once it starts, so the producer/consumer imbalance is easy to see. The provider is only the reproduction vehicle here - the fix is in the core player and benefits all providers.

Related: #5302 (closed - same gapless idea, scoped to one provider), #5732 (open - adds a provider stop() callback to the TTS controller; that would also close the "stale audio after Stop" gap on the provider side).

Priority

Medium (Would be very useful)

Are you willing to test this on staging/unstable branch if this is implemented?

Yes