The model was never the problem, and that is exactly why the bug took three days to find.
My ticket-classification service started returning the fallback label for long, non-English messages shortly after I moved the inference path to a cheaper endpoint, and every instinct pointed at the new model.
The real culprit was a token-counting mismatch that silently truncated the prompt before the model ever saw the classification instruction.
The Symptom The failure was remarkably consistent, which made it even more misleading.
Messages under roughly two thousand characters classified correctly, while longer ones, especially in German and Japanese, fell through to a generic "other" bucket with a perfectly valid JSON response.
The parser was not the issue, the prompt had not changed in weeks, and the retry logic never fired because the endpoint returned a normal 200 status.
My first assumption was that the cheaper model was simply weaker at long-context reasoning, so I ran a controlled comparison using the same fifty tickets against the previous endpoint.
The old path classified all fifty correctly, the new one failed on nineteen, and that result seemed to confirm the model-quality theory.
What bothered me was the distribution: the failures clustered exactly where the input length crossed a threshold, and no ticket under that threshold ever failed.
The Reproduction To isolate the variable, I needed a clean environment where I could swap endpoints without touching the production deployment, and MonkeyCode's free server option turned out to be a practical debugging tool.
The project is open source, and its free model access let me replay the failing tickets without spending my own quota, so I spun up a disposable instance and pointed the same harness at the same prompt.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The reproduction took about twenty minutes, and the result was identical on every retry: long inputs failed, short inputs passed.
The Root Cause The breakthrough came when I logged the token count of the incoming payload instead of the character count.
My client code had a hard character limit that was supposed to keep every prompt inside the context window, but the tokenizer used by the new endpoint split German and Japanese text into roughly twice as many tokens per character as English.
The client-side truncation then cut the message at the character boundary, which happened to fall right before the classification instruction, so the model produced a confident fallback with no idea that the instruction had ever existed.
The Fix The fix had three parts, and none of them involved changing the model.
I replaced the character-based guard with a token-based guard that used the same tokenizer as the inference endpoint, so the length check and the actual consumption could never disagree again.
I added a sentinel instruction that asked the model to include a marker in every response, and I rejected any output that lacked the marker, which turned silent truncation into a loud validation error.
Finally, I added a regression test that fed the harness a set of long, non-English fixtures and asserted that the marker always appeared.
Here is the guard that fixed the production bug, and it is short enough to review in one sitting: The key detail is that truncation happens on the message side, never on the instruction side, and the budget reserves a safety margin for the model's own output tokens.
If you are replaying a production incident, the reproduction script is even simpler, and it only needs the endpoint URL and a long multilingual string: If that call returns a valid JSON object without the DONE marker, you have reproduced the exact class of bug, and the fix is the guard above.
The sentinel marker deserves a moment of explanation because it looks redundant at first glance.
The model was already returning valid JSON, so a JSON parser alone would never catch the truncation, and the marker lives insi