LlamaCppTokenizer drops every token whose piece exceeds the 32-byte buffer: llama_token_to_piece reports an undersized buffer with a negative return, so the n > size retry is unreachable

Describe the issue as clearly as possible:

LlamaCppTokenizer.__init__ builds its fallback vocabulary (the branch taken when model.tokenizer_ has no hf_tokenizer) by calling llama_token_to_piece with a fixed 32-byte buffer. When a token's piece needs more than 32 bytes, llama.cpp does not return a value greater than the buffer size — it returns the negative required size. The loop tests if n < 0: continue first, so every token whose piece exceeds 32 bytes is skipped as an "invalid token" and never enters self.vocabulary. The if n > size: retry immediately below — whose own comment reads "n > size means the piece was truncated; retry with a larger buffer so distinct tokens are not collapsed" — is therefore unreachable dead code.

Because the _hf_tokenizer branch keeps every token from self._hf_tokenizer.get_vocab(), the two branches build vocabularies with different coverage, and the FSM tokenizer can never emit the dropped ids.

Root cause: src/outlines/models/llamacpp.py:67 — if n < 0: continue runs before the if n > size: retry at line 71, but an undersized buffer is signalled as n == -needed (negative), never as n > size. So the guard intended for invalid tokens swallows the truncation signal instead.

Related issues/PRs: #1819 (closed) and its merged fix #1820 — #1820 is the PR that introduced this behaviour: it replaced the pre-fix token_piece = buffer[:n] (which at least kept the truncated token) with if n < 0: continue plus the retry that can never fire. #1718 is an open meta tracker ("Review/improve the custom tokenizers for transformers and llamacpp") that never mentions long pieces or the negative return. No open issue or PR covers this defect.

This report was prepared with AI assistance and reviewed by a human, per the "AI-assisted contributions" section of CONTRIBUTING.md.

Steps/code to reproduce the bug:

# repro_llamacpp_longpiece.py
"""Offline repro: fake llama_cpp module that mimics llama.cpp's real return contract."""
import ctypes
from unittest.mock import MagicMock, patch

from outlines.models.llamacpp import LlamaCppTokenizer

LONG = "a" * 40          # 40-byte piece, > the hardcoded 32-byte buffer
PIECES = {0: "hello", 1: " world", 2: "</s>", 3: LONG, 4: "bye"}
calls = []


def fake_llama_token_to_piece(vocab, token, buf, buf_size, lstrip, special):
    # verbatim semantics of llama.cpp llama_vocab::impl::token_to_piece
    # (src/llama-vocab.cpp): "if (length < (int32_t) size) return -(int32_t) size;"
    data = PIECES[token].encode()
    calls.append((token, buf_size, len(data)))
    if buf_size < len(data):
        return -len(data)
    ctypes.memmove(buf, data, len(data))
    return len(data)


def make_tokenizer():
    model = MagicMock()
    model.token_eos.return_value = 2
    model.n_vocab.return_value = len(PIECES)
    del model.tokenizer_          # llama_cpp default: tokenizer_ has no hf_tokenizer
    with patch.dict("sys.modules", {"llama_cpp": MagicMock(
        llama_model_get_vocab=MagicMock(return_value=MagicMock()),
        llama_token_to_piece=fake_llama_token_to_piece,
    )}):
        return LlamaCppTokenizer(model)


tok = make_tokenizer()
print("n_vocab() reported by the model      :", len(PIECES))
print("token ids kept in tokenizer.vocabulary:", sorted(tok.vocabulary.values()))
print("token ids dropped                    :",
      sorted(set(PIECES) - set(tok.vocabulary.values())))
print("piece for id 3 (len %d) is in vocab   :" % len(LONG),
      LONG in tok.vocabulary)
print("calls (token_id, buf_size, needed_len):", calls)
retried = [c for c in calls if c[0] == 3]
print("times id 3 was asked of llama.cpp     :", len(retried),
      "(a retry with a larger buffer would show buf_size >= 40)")
print("any buffer larger than the initial 32 :",
      sorted({c[1] for c in calls}))

No model weights are needed; the fake reproduces llama.cpp's documented return contract exactly.

Against a real vocabulary the same defect is measurable directly — iterate a Qwen3-0.6B GGUF's token ids, call llama_token_to_piece with a 32-byte buffer, and compare the kept ids with LlamaCppTokenizer(model).vocabulary.

Expected result:

The token should be present in vocabulary, with the full (untruncated) piece text:

token ids kept in tokenizer.vocabulary: [0, 1, 2, 3, 4]
token ids dropped                    : []
piece for id 3 (len 40) is in vocab   : True

The concrete basis is llama.cpp's own contract and its own caller. llama_vocab::impl::token_to_piece (src/llama-vocab.cpp) reports an undersized buffer by returning the negative required size and never a positive value greater than the buffer:

auto _try_copy = [=] (const char * token, size_t size) -> int32_t {
    ...
    if (length < (int32_t)size) {
        return -(int32_t) size;
    }
    memcpy(buf, token, size);
    return (int32_t) size;
};

where length is the caller's buffer size. Every other return in that function is 0 (suppressed/unknown token), -(int)result.size() (RWKV) or -1 (PLaMo2); no path returns a positive value greater than length. llama.cpp's own consumer handles exactly this shape — llama_vocab::impl::token_to_piece_for_cache:

if (n_chars < 0) {
    piece.resize(-n_chars);
    int check = vocab.token_to_piece(...);
    ...
}

llama-cpp-python exposes llama_token_to_piece as a raw c_int32 ctypes binding with no resize wrapper, so -(required size) reaches Python verbatim. The repo's own comment at lines 69-70 states the intended behaviour.

Note also that tests/models/test_llamacpp_tokenizer.py::test_vocab_truncation_retry_path encodes the wrong contract: its fake_llama_token_to_piece returns the full positive length (40) for a 32-byte buffer, which llama.cpp never does, so the test passes while the token is dropped in reality.

Error message:

Offline repro (python repro_llamacpp_longpiece.py), verbatim:

n_vocab() reported by the model      : 5
token ids kept in tokenizer.vocabulary: [0, 1, 2, 4]
token ids dropped                    : [3]
piece for id 3 (len 40) is in vocab   : False
calls (token_id, buf_size, needed_len): [(0, 32, 5), (1, 32, 6), (2, 32, 4), (3, 32, 40), (4, 32, 3)]
times id 3 was asked of llama.cpp     : 1 (a retry with a larger buffer would show buf_size >= 40)
any buffer larger than the initial 32 : [32]

The same run against a real Qwen3-0.6B vocabulary, verbatim:

real Qwen3-0.6B vocab size            : 151643
pieces needing > 32 bytes              : 1401
ids missing from tokenizer.vocabulary : 1401
...of those, long-piece ids dropped : 1401
buffer sizes ever passed to the C API : [32]
n>size returns seen                   : 0
negative returns seen / of which kept : 1401 0
AssertionError: BUG: 1401 long-piece ids were dropped

n > size is never observed (it cannot be), and every negative return is discarded. Real vocabularies contain such tokens — long - / Ġ runs, e.g. a 128-byte Ġ*128 piece.

Outlines/Python version information:

1.3.4.dev2+gc52af8472  (commit c52af8472c6fe8a607a733ec11b709476af77a96)
Python 3.12.14 (main, Aug 25 2026, 13:50:33) [Clang 22.1.3 ]
macOS-26.6.2-arm64-arm-64bit
llama-cpp-python 0.3.35
transformers 5.17.0
numpy 2.5.3
pydantic 2.13.5

Context for the issue:

Reachable from any from_llamacpp(Llama(...)) / LlamaCpp(model) where model.tokenizer_ has no hf_tokenizer — the llama-cpp-python default (self.tokenizer_ = tokenizer or LlamaTokenizer(self)), and the path tests/models/test_llamacpp_tokenizer.py forces with del model.tokenizer_. Those ids are absent from vocabulary, so structured generation through this adapter can never select them, and any grammar/regex that would require such a piece cannot be satisfied.

I can open a PR that treats a negative return as the required buffer size and retries once, matching llama.cpp's own token_to_piece_for_cache handling, and corrects test_vocab_truncation_retry_path so its fake follows the real return contract.