Self-hosted models via openai-compatible fall back to maxOutputTokens=4096 and supportsStructuredOutput=false

Summary

Model capabilities are resolved from a lookup keyed on the model name string (this.modelId). It recognises hosted models (claude-*, gemini-*, deepseek-v4, mistral.*, …) but has no entry for open models such as Qwen. Any unrecognised model falls through to:

return { maxOutputTokens: 4096, supportsStructuredOutput: false, rejectsSamplingParameters: false, isKnownModel: false }

For self-hosters using DEFAULT_LLMS=openai-compatible:<model>, this silently applies two conservative limits regardless of what the backend actually supports.

Impact

1. Output capped at 4096 tokens. Longer generations are truncated (finish_reason: length) even when the server allows far more. Our vLLM instance serves a 262,144-token context and, with no max_tokens supplied, happily generates well past 4096 and stops naturally.

2. Structured output disabled. With supportsStructuredOutput: false, requests fall back to prompt-based "return JSON" instructions rather than native constrained decoding. Our backend supports response_format: json_schema and returns valid schema-conformant JSON when asked directly. Prompt-based JSON works most of the time, but it is strictly less reliable than server-enforced decoding for a classification path that runs on every email.

Neither limit is visible in the UI, and neither can be worked around externally — the capability decision happens before the request is built, so a proxy in front of the backend can't influence it.

Reproduce

1. Serve any open model behind an OpenAI-compatible endpoint (vLLM, SGLang, etc.)
2. Set DEFAULT_LLMS=openai-compatible:<model-name> and OPENAI_COMPATIBLE_BASE_URL
3. Ask for a long response — it stops at 4096 output tokens
4. Observe that requests do not use native structured output

Suggested fix

Any of:

• Add entries for common open models (Qwen, Llama, Mistral open weights, GLM, DeepSeek open weights)
• Allow explicit overrides via env, e.g. OPENAI_COMPATIBLE_MAX_OUTPUT_TOKENS and OPENAI_COMPATIBLE_SUPPORTS_STRUCTURED_OUTPUT
• Treat openai-compatible as capability-opt-in rather than assuming the most conservative defaults, since the operator knows their own backend

The env-override option is probably most robust, since self-hosted backends vary too much to enumerate. Happy to open a PR if you'd like a particular approach.

Environment

• Self-hosted, Docker Compose
• Backend: vLLM serving an open MoE model, OpenAI-compatible API, 262k context