#3595·sure

Bug: LLM JSON mode `Auto` does not fall back to `none` when the strict response cannot be parsed

Author: joshiarnavCreated Sep 16, 2026Updated Sep 16, 2026

How are you using Sure?

  • I am a self-hosted user (local only)

Self hoster checklist

  • Self hosted app commit SHA: reproduced against main @ 44bbfb2
  • I have confirmed that my app's commit is the latest version of Sure
  • Where are you hosting? Other: LXC (Proxmox) with a local OpenAI-compatible model (Ollama / llama.cpp)

Bug description

Auto JSON mode (the default for custom providers) is documented as "try strict json_schema, fall back to no constraint when the model can't cope". Today the fallback fires in exactly two cases:

  1. the provider answers HTTP 400 to the strict request (rescue Faraday::BadRequestError in auto_categorize_openai_generic), or
  2. the strict request parses fine but >50% of results are null/missing (auto_categorize_with_auto_mode).

It does not fire when the strict request returns HTTP 200 but the body cannot be turned into categorizations. extract_categorizations_generic / parse_json_flexibly raise Provider::Openai::Error ("Could not parse JSON from response: …" or "Could not find categorizations in response"), and that exception propagates straight out of auto_categorize_with_auto_mode past the null-ratio heuristic. So Auto behaves exactly like Strict in the failure mode where a compatibility fallback would help most.

This is common with local reasoning models: the model spends its output budget thinking, the strict JSON is truncated ({"categorizations":[{"transaction_id":…), the HTTP call "succeeds", parsing fails, and the whole batch fails instead of retrying without response_format.

The same structure exists in Provider::Openai::AutoMerchantDetector (auto_detect_merchants_with_auto_mode).

To Reproduce

Against an OpenAI-compatible endpoint that returns a well-formed HTTP 200 whose choices[0].message.content is not parseable JSON (easiest: stub client.chat in a test, or point at a small local model with a low num_predict):

  1. Set JSON mode to Auto (default) in Settings → Self-hosting → OpenAI.
  2. Run "Auto-categorize" on a batch of transactions.
  3. Observe a single request and a failed job with Could not parse JSON from response / Could not find categorizations in response. No none-mode retry is attempted.

Expected behavior

In Auto mode only: if the strict attempt returns a response that cannot be parsed into the expected categorization structure, retry once with JSON_MODE_NONE (same as the HTTP-400 path). Network, auth, timeout and other provider errors must not be converted into retries. Strict and None mode behaviour unchanged.

Proposed fix

Introduce a narrow Provider::Openai::ResponseFormatError < Provider::Openai::Error raised by the parse/shape helpers, rescue only that around the strict attempt inside *_with_auto_mode, retry with JSON_MODE_NONE once, and let a failure of the retry surface normally. Apply to both AutoCategorizer and AutoMerchantDetector.