Chatterbox Turbo: 10 of 19 paralinguistic tags are tokenized correctly but have no audible effect

Chatterbox Turbo: 10 of 19 paralinguistic tags are tokenized correctly but have no audible effect

Package: mlx-audio==0.4.5 Model: chatterbox_turbo (mlx-community/chatterbox-turbo-4bit, also reproduced on -8bit) Platform: macOS, Apple Silicon (M4 Pro)

Summary

ResembleAI/chatterbox-turbo ships 19 paralinguistic tags in added_tokens.json (ids 50257–50275). Under mlx-audio, the 9 "sound" tags produce audible output, while the 10 "emotion/delivery" tags produce no audible change at all — no whisper, no anger, no effect of any kind. They are silently ignored.

Working (9): [laugh] [chuckle] [gasp] [cough] [sigh] [groan] [sniff] [shush] [clear throat]

No effect (10): [whispering] [angry] [fear] [surprised] [crying] [happy] [sarcastic] [dramatic] [narration] [advertisement]

Users report all 19 working under the ONNX runtime (resemble-ai/chatterbox#492), which suggests the tags themselves are trained and the divergence is in this implementation.

Expected behaviour

[whispering] There is someone in the hallway. should be rendered as whispered speech, as the other 9 tags render their respective sounds.

Actual behaviour

Rendered identically to the same sentence with the tag removed. The tag is not spoken aloud (confirming it is consumed as a token, not as literal text) — it simply has no effect on the output.

Reproduction

import os
os.environ.setdefault("HF_HOME", "./models")
from mlx_audio.tts.utils import load_model
import numpy as np, soundfile as sf

model = load_model("mlx-community/chatterbox-turbo-4bit")

for tag in ["", "[whispering] ", "[angry] ", "[laugh] "]:
    audio = None
    for r in model.generate(f"{tag}There is someone in the hallway.", stream=False):
        audio = r.audio
    name = (tag.strip("[] ") or "plain")
    sf.write(f"out-{name}.wav", np.asarray(audio, dtype=np.float32).reshape(-1), 24000)

out-laugh.wav differs audibly from out-plain.wav. out-whispering.wav and out-angry.wav are indistinguishable from out-plain.wav.

Ruled out

Tokenization. All 19 tags resolve to single token ids via the loaded tokenizer, and inference uses that same tokenizer (chatterbox_turbo.py:916, not the ord() fallback at :925):

tokenizer class: GPT2Tokenizer
vocab size     : 50276

[laugh]            -> [50275]   SINGLE TOKEN
[chuckle]          -> [50274]   SINGLE TOKEN
[gasp]             -> [50273]   SINGLE TOKEN
[cough]            -> [50270]   SINGLE TOKEN
[sigh]             -> [50268]   SINGLE TOKEN
[groan]            -> [50271]   SINGLE TOKEN
[sniff]            -> [50272]   SINGLE TOKEN
[shush]            -> [50269]   SINGLE TOKEN
[clear throat]     -> [50267]   SINGLE TOKEN
  => 9/9 recognised

[whispering]       -> [50260]   SINGLE TOKEN
[angry]            -> [50257]   SINGLE TOKEN
[fear]             -> [50258]   SINGLE TOKEN
[surprised]        -> [50259]   SINGLE TOKEN
[crying]           -> [50264]   SINGLE TOKEN
[happy]            -> [50265]   SINGLE TOKEN
[sarcastic]        -> [50266]   SINGLE TOKEN
[dramatic]         -> [50262]   SINGLE TOKEN
[narration]        -> [50263]   SINGLE TOKEN
[advertisement]    -> [50261]   SINGLE TOKEN
  => 10/10 recognised

Note the split is exactly contiguous: the 9 that work are ids 50267–50275, the 10 that don't are 50257–50266.

Vocabulary truncation. t3_config.py sets text_tokens_dict_size: 50276 and models/t3/gpt2.py sets vocab_size: 50276, so the embedding covers all 19 ids.

Quantization. Reproduced identically on chatterbox-turbo-4bit and chatterbox-turbo-8bit. (fp16 not yet tested.)

Reference-audio conditioning. Reproduced with a cloned reference wav and with the checkpoint's built-in voice.

CFG / exaggeration. Not applicable — chatterbox_turbo.py:862 explicitly logs that CFG, min_p and exaggeration are unsupported by Turbo and ignored, so no sampling parameter is available to amplify the conditioning.

punc_norm. Does not alter bracketed text: the leading [ is not lowercase, so the capitalisation branch does not fire, and no replacement rule touches [ or ].

Possibly relevant

t3.inference_turbo is called with temperature=0.8, top_k=1000, top_p=0.95, repetition_penalty=1.2 (chatterbox_turbo.py:931), and S3Gen runs with a hardcoded n_cfm_timesteps=2 (:953). I have not established that any of these is the cause — noted only because they are the parameters that differ from a reference implementation's defaults.

Still to confirm

I have not yet A/B'd against the reference PyTorch implementation (chatterbox.tts_turbo.ChatterboxTurboTTS) on the same checkpoint and text. If that also fails to whisper, the problem is upstream of mlx-audio and this report should be redirected.

References

• Tag list: https://huggingface.co/ResembleAI/chatterbox-turbo/blob/main/added_tokens.json
• Resemble maintainer lists only the 9 sound tags: https://huggingface.co/ResembleAI/chatterbox-turbo/discussions/21
• ONNX users report all 19 working: https://github.com/resemble-ai/chatterbox/issues/492