I run a small local model — Qwen3.5 4B via Ollama — as one deterministic-checked step inside some tooling.
This is what still reproduces today, on named versions, with the scripts to check it yourself.
Where a result did not survive re-running, that is here too, because that turned out to be the most useful part.
Versions, so you can pin them: Ollama 0.32.6, model , digest , Q4_K_M.
None of this is a new technique.
The underlying pattern is Best-of-N with verifiers, which is well established.
These are measurements inside a known pattern. can hand you an empty answer With thinking enabled, the chain-of-thought and the final answer share one token budget.
When the thinking consumes it, comes back empty while holds the content.
Code that reads only sees and concludes the call failed.
It didn't.
Measured today, one-line question, : Fix: for extraction and structured tasks, read both fields, or turn thinking off.
Whether it triggers depends on your budget — tighten to make it likely. (Separately: there is evidence that pushing long chain-of-thought onto small models can hurt accuracy — Luo et al., "Through the Valley: Path to Effective Long CoT Training for Small Language Models," EMNLP
2025.
That is a training-time effect, not this API field-routing quirk.
Different problem; noting it so the two don't get conflated.) Self-critique made the 4B worse I asked it to critique its own answer.
It verbally retracted the wrong fix — "Remove the check proposed above" — and then re-emitted that same wrong fix in code, in the same reply.
One anecdotal example of what Huang et al. measure at scale in "Large Language Models Cannot Self-Correct Reasoning Yet" (arXiv:2310.01798, ICLR 2024): on GSM8K, GPT-4's own accuracy drops 95.5% → 91.5% → 89.0% under intrinsic self-critique.
Self-critique is not a free reliability layer.
You need an external judge, not the model grading itself.
Request precision beat model depth Same input, two prompts.
Asking for output keys that match the source's naming literally moved accuracy 7/10 → 10/10.
A numeric spec — "each row has exactly 3 cells and 4 dashes" — fixed a table that a descriptive spec — "a well-formed separator row" — kept producing broken.
Well-worn prompt-engineering advice.
Included only because I have the before/after on the same input.
The external judge needs its own negative control Here is the part I got wrong for a while, and it is the expensive one. "Put a deterministic external check on the output" is half an instruction.
I had that check.
It passed a fabrication.
A merge task, two columns into one, 81 rows.
The model invented 5 values that were never in the input.
The check returned , , and exit code
0.
It was not bypassed and it did not crash.
Every invented value was drawn from a closed vocabulary — every value that could appear did appear somewhere in the input — so every token-level assertion the check ran was satisfied.
The judge was asked a question that could not tell the two cases apart.
So the rule I would give someone starting this: A judge that has never been made to fail has not been shown to pass.
Build the check, then feed it material it must reject.
If it stays green, you don't have a judge — you have a second thing that agrees with the model.
And when the output is a pure function of the input, compare it to the input mechanically before accepting it.
An exit code is not evidence.
And one result that did not survive An earlier note of mine reported that this model drops the space at a Latin↔Arabic boundary — ask for , get back .
I had it in 2 of 2 runs and treated it as a real defect.
Re-running it today: 53 generations.
Zero reproductions.
Seven prompt shapes on the pinned model, six of those shapes repeated on two other local 4B variants, plus the original echo prompt.
I can't tell you it was fixed, and I'm not going to pretend otherwise — the model file on disk is the same one that was there when I first measured it.
Prompt shape doesn't explain it either; I tried the