I benchmarked a vision-language model and scored it at 0.31.
The real number was 0.70.
Same model, same weights, same hardware, same 100 questions.
The only thing that changed was how I read its output.
I had already written up the 0.31 as a capability finding and concluded the model was unsuitable.
That conclusion was wrong, and the failure was entirely in my harness.
Here is the mistake, because I doubt I am the only one making it.
The setup I was evaluating a batch of open-weight and frontier models on a multiple-choice benchmark: multi-view driving scenes, four options per question, one correct answer.
Standard stuff.
The prompt asked for reasoning followed by a final line, .
My scoring code did the obvious thing: That last comment is the bug.
What actually happened The model I was testing is a "thinking" model.
It emits a long internal reasoning trace before it commits to an answer.
I had a generation budget of 1024 tokens.
On easy questions it reasoned briefly, emitted , and scored fine.
On hard questions it reasoned at length, hit the token cap mid-thought, and never emitted the answer line at all.
So the harness scored every one of those as wrong. 64 of 100 questions returned no parseable answer.
Zero of those were image-loading errors or crashes.
They were all truncation.
And the truncation was not random: Look at that distribution.
The questions the model failed to answer were precisely the questions that required the most reasoning.
My harness was systematically discarding the model's performance on exactly the hard subset I was trying to measure, and reporting the result as a capability ceiling.
Of the 36 it did answer, it got 86% right.
The model was fine.
My measurement was garbage.
The fix Stop parsing free text.
Constrain the decoding to a schema.
With Ollama: The equivalent exists nearly everywhere now: with a JSON schema on OpenAI-compatible endpoints, structured outputs in vLLM via , Outlines, or plain grammar-constrained sampling in llama.cpp.
The point is that the model can no longer produce an unparseable output.
The constraint is applied during sampling, not checked afterwards.
Re-ran the same 100 questions: 0.70, with a 100% answer rate.
The categories that had been at zero came back at 0.82 and 0.75.
The second-order trap I hit a related version of this on a larger model in the same family, and the fix was less obvious.
That model also truncated, on 36 of 100 questions.
I had a fallback that recovered an answer letter from the tail of a truncated trace, so those rows produced something.
The score came out 0.67.
But the recovered answers were junk in a specific, dangerous way.
Predicted answers skewed heavily toward option A (48 predictions of A against 23 in the ground truth).
When the model gets cut off mid-reasoning, the letter you scrape from the tail is not a decision, it is whatever token happened to be nearby.
That is position bias, and it looks exactly like a real answer to your scorer.
Rows where the reasoning actually completed scored 0.83.
I re-ran just the truncated rows with a budget of 8192 instead of 3072, and the merged score came to 0.73.
So: tail-recovery is worse than a non-answer.
A non-answer is visibly missing.
A biased recovered answer silently contaminates your accuracy in a direction you did not choose.
Note also that 16 of those 36 questions still truncated at 8192 tokens. 0.73 remains a floor, not a measurement.
What I do now Match the evaluator's generation budget to the training or intended-use budget.
Mismatched budgets scored 31% of my rows incorrectly in one run.
If you fine-tuned at 512 tokens, do not evaluate at
128.
Track truncation as a first-class metric.
Log and next to accuracy.
If either moves, your accuracy number is not comparable to the previous run.
I now fail a run outright if the parse-failure count is anything other than what I expect.
Distinguish infrastructure failure from poor performance.
An unparseable output is a harness event, not a model event.
They