mimi uses the interleaved RoPE convention but the checkpoints are stored in the transformers layout

The mimi transformer applies the interleaved (GPT-J style) rotary embedding, but the kyutai/mimi checkpoint that Model::new loads is stored in the transformers layout, which rotates the two halves of the head dimension (NeoX style). Attention scores are therefore computed against the wrong frequency pairing at every position but the first, and both the encoder and the decoder drift away from the reference implementation as the sequence grows.

https://github.com/huggingface/candle/blob/c30b26cd3bb59e5b39927eebff00884b0c0729d5/candle-transformers/src/models/mimi/transformer.rs#L82

Transformers uses rotate_half for this model, models/mimi/modeling_mimi.py, which is what candle_nn::rotary_emb::rope implements.

Impact

Everything that goes through models::mimi, which is the mimi and csm examples. Encoded codes are wrong and decoded audio is measurably degraded, but it is a quiet failure. Audio still sounds like speech, and nothing errors out.

Reproduction

A 5.04 s mono clip at 24 kHz, 63 frames, f32 on CPU, candle at d5fee525, where this file is identical to the permalink above.

Codes from transformers 4.57.3:

import torch, soundfile as sf
from transformers import MimiModel

wav, _ = sf.read("audio.wav", dtype="float32")            # mono, 24 kHz
m = MimiModel.from_pretrained("kyutai/mimi", dtype=torch.float32).eval()
with torch.no_grad():
    codes = m.encode(torch.from_numpy(wav)[None, None, :], num_quantizers=16).audio_codes

Codes from candle, same checkpoint, same audio:

cargo run --example mimi --features mimi -r -- --cpu audio-to-code audio.wav codes.safetensors

359 of the 1008 codes match, and not one of the 63 frames matches in full. Agreement is highest in the first frame, 14 of its 16 codes, and falls off immediately after, which is the signature of a position-dependent difference rather than a numerical one.

Feeding the transformers codes back through Model::decode and comparing samples with MimiModel.decode:

Comparing codes is the cleaner check of the model itself, since the mimi example applies normalize_loudness before writing its wav.