Bug: LLM JSON mode `Auto` does not fall back to `none` when the strict response cannot be parsed
How are you using Sure?
- I am a self-hosted user (local only)
Self hoster checklist
- Self hosted app commit SHA: reproduced against
main@ 44bbfb2 - I have confirmed that my app's commit is the latest version of Sure
- Where are you hosting? Other: LXC (Proxmox) with a local OpenAI-compatible model (Ollama / llama.cpp)
Bug description
Auto JSON mode (the default for custom providers) is documented as "try strict json_schema, fall back to no constraint when the model can't cope". Today the fallback fires in exactly two cases:
- the provider answers HTTP 400 to the strict request (
rescue Faraday::BadRequestErrorinauto_categorize_openai_generic), or - the strict request parses fine but >50% of results are null/missing (
auto_categorize_with_auto_mode).
It does not fire when the strict request returns HTTP 200 but the body cannot be turned into categorizations. extract_categorizations_generic / parse_json_flexibly raise Provider::Openai::Error ("Could not parse JSON from response: …" or "Could not find categorizations in response"), and that exception propagates straight out of auto_categorize_with_auto_mode past the null-ratio heuristic. So Auto behaves exactly like Strict in the failure mode where a compatibility fallback would help most.
This is common with local reasoning models: the model spends its output budget thinking, the strict JSON is truncated ({"categorizations":[{"transaction_id":…), the HTTP call "succeeds", parsing fails, and the whole batch fails instead of retrying without response_format.
The same structure exists in Provider::Openai::AutoMerchantDetector (auto_detect_merchants_with_auto_mode).
To Reproduce
Against an OpenAI-compatible endpoint that returns a well-formed HTTP 200 whose choices[0].message.content is not parseable JSON (easiest: stub client.chat in a test, or point at a small local model with a low num_predict):
- Set JSON mode to
Auto(default) in Settings → Self-hosting → OpenAI. - Run "Auto-categorize" on a batch of transactions.
- Observe a single request and a failed job with
Could not parse JSON from response/Could not find categorizations in response. Nonone-mode retry is attempted.
Expected behavior
In Auto mode only: if the strict attempt returns a response that cannot be parsed into the expected categorization structure, retry once with JSON_MODE_NONE (same as the HTTP-400 path). Network, auth, timeout and other provider errors must not be converted into retries. Strict and None mode behaviour unchanged.
Proposed fix
Introduce a narrow Provider::Openai::ResponseFormatError < Provider::Openai::Error raised by the parse/shape helpers, rescue only that around the strict attempt inside *_with_auto_mode, retry with JSON_MODE_NONE once, and let a failure of the retry surface normally. Apply to both AutoCategorizer and AutoMerchantDetector.
Source: we-promise/sure