litellm route sends json_schema that z.ai ignores; native zai provider succeeds
Setup: self-hosted Hindsight 0.10.0; retain lane HINDSIGHT_API_RETAIN_LLM_PROVIDER=litellmrouter with a member openai/glm-5.3-flash at https://api.z.ai/api/coding/paas/v4 (z.ai coding plan, OpenAI-compatible).
Observed: litellm_llm.py always sends response_format: {"type":"json_schema", …} (line ~331). The z.ai endpoint ignores json_schema and answers in prose (1. what="…", when="…"), or a bare array with strict: true. Over 24 h on ~625 extraction calls: 109 LiteLLM returned invalid JSON, retrying... and 37 Fact extraction failed: … JSONDecodeError (documents lost). Measured offline with the exact prompts/preview retain prompt: 1/8 first-attempt valid through that route, 8/8 for the same model through OpenRouter.
Native provider works: HINDSIGHT_API_RETAIN_LLM_PROVIDER=zai uses openai_compatible_llm.py's soft enforcement (schema in prompt + json_object, fence stripping) → 16/16 valid in our re-test, and the live retry warnings stopped after switching.
Ask: let the LiteLLM route fall back to the soft json_object path (or honour the provider's declared capability) when a backend ignores json_schema, or document that z.ai must use the native provider. Happy to share the measurement scripts.
Source: vectorize-io/hindsight