GGUF backend's echo=true handling doesn't work against current llama-server — every loglikelihood request fails

GGUF backend's echo=true handling doesn't work against current llama-server — every loglikelihood request fails

File: lm_eval/models/gguf.py

GGUFLM.gguf_completion() sends echo=True, max_tokens=1 and expects the response to contain logprobs for the entire echoed prompt plus continuation, then get_result() walks logprobs["text_offset"] to find where the continuation starts:

def gguf_completion(self, context, continuation=None, ...):
    ...
    if continuation:
        prompt += continuation
        request.update({"prompt": prompt, "max_tokens": 1, "echo": True})
    ...

def get_result(logprobs, context_length):
    offsets = logprobs["text_offset"]
    tokens = logprobs["tokens"]
    tokens_logprobs = logprobs["token_logprobs"]
    idx = 0
    while offsets[idx] < context_length:
        idx += 1
    ...

Against a current llama-server build (b9847, and I'd expect any recent one), this breaks in two independent ways:

1. Schema mismatch. llama-server's OpenAI-compatible endpoint returns the newer logprobs.content: [{token, logprob, top_logprobs}, ...] shape, not the legacy text_offset / tokens / token_logprobs / top_logprobs-as-dict shape this code expects. Every response fails the "token_logprobs" in logprobs check in loglikelihood().
2. echo=true doesn't actually echo. Even after working around (1), echo=true on current llama-server does not return logprobs — or even text — for the prompt tokens. A request with max_tokens=1, echo=true against a long prompt returns exactly one logprobs.content entry (the newly generated token) and a text field containing only that new token, not the prompt. So even with a schema-compatible parser, there's no way to recover log P(continuation | context) via this endpoint the way the code assumes.

Reproduction

# any llama-server serving an OpenAI-compatible /v1/completions endpoint
curl -s http://127.0.0.1:8080/v1/completions -H "Content-Type: application/json" -d '{
  "prompt": "Question: What is the capital of France?\nAnswer: Paris",
  "logprobs": 10, "temperature": 0, "max_tokens": 1, "echo": true
}'
# -> logprobs.content has length 1 (only the new token), and .text is just the new token's text

lm_eval --model gguf --model_args base_url=http://127.0.0.1:8080 --tasks arc_easy --limit 50
# -> every request logs: WARNING [models.gguf:94] Invalid logprobs data. Expected 'logprobs' to contain 'token_logprobs' list.
# -> ValueError: zip() argument 2 is longer than argument 1  (evaluator.py, because loglikelihood() silently dropped every failed request instead of raising)

Workaround I used

Force the exact continuation through a GBNF grammar (root ::= "<continuation text>") instead of relying on echo, then sum the logprobs.content[].logprob values from that forced generation. Mathematically the same quantity (log P(continuation | context) under temperature 0), just produced via constrained generation instead of echo:

grammar = f'root ::= "{escaped_continuation}"'
request = {"prompt": context, "grammar": grammar, "logprobs": 1, "temperature": 0.0, "max_tokens": ...}
# sum response["choices"][0]["logprobs"]["content"][*]["logprob"]

This got me a working arc_easy run end to end. Happy to open a PR with a patched gguf.py that (a) supports the new logprobs.content schema for backward/forward compatibility, and (b) uses the grammar-forced-continuation approach instead of echo when talking to a server that doesn't support it, if that's a direction the maintainers want. Filing the report first in case there's already a known plan here (or a reason echo is expected to keep working that I'm missing on my server config).

Context

Found this while trying to get a real accuracy number for a hackathon submission (ADTC 2026) whose reference profiler tool (adtc-profiler) shells out to this exact code path and was silently returning empty accuracy results because of it — filed a related but separate issue there: Africa-Deep-Tech-Foundation/adtc-profiler#4.