Model-tier guidance for Khoj's intermediate steps and automations — offering a benchmark, plus a question about auto-populating chat models via /v1/models

Hi Khoj team,

Khoj's staying power is deserved — nearly five years of continuous iteration in a space where most personal-AI projects fade within eighteen months. The "second brain" framing is the right one: the value isn't the chat box, it's that documents, web research, agents, automations, and every entry point (Obsidian, Emacs, WhatsApp, desktop, phone) live in one self-hostable system with a real Django admin panel behind it. The automated research workflows with scheduled newsletters are quietly the most differentiated feature — almost nobody else ships unattended recurring research out of the box.

Two cost-shape observations from running mixed-model setups

The intermediate steps burn tokens nobody sees. Khoj's docs are clear that the default chat model is used for "intermediate steps like intent detection, web search" and that the ServerChatSettings' Default and Summarizer fields exist to control this — which tells me the team already understands the problem. But in practice, most self-hosters set one capable model and forget it, and every one of their scheduled research runs, notification generations, and summarization passes runs through it. An automation that fires daily does its intent detection and synthesis on the same model that writes the final newsletter. The docs currently say which settings control this but not what quality each step actually needs — and that's a data question, not a config question.

The admin-panel per-model tax. Khoj's official use-openai-proxy documentation already blesses the gateway pattern (the AI Model API entry has a base URL field described as "only relevant if you're using another OpenAI-compatible proxy server") — so the concept needs no introduction here. But the workflow that follows has real friction: every model a user wants must be manually registered as its own Chat Model entry, with model-type, a vision flag, and optional tokenizer and max-prompt-size fields the docs themselves hedge on ("set them only if you're sure... contact us if you're unsure"). A user who wants a flagship for research, a cheap fast model for intent detection and summarization, and a bilingual model for Chinese-language notes is creating three entries against two or three provider accounts — each new model another row with fields to guess at. The model list itself is the thing users have to maintain by hand.

Why I'm writing

I maintain AnyLLM, an OpenAI-compatible API gateway that fronts 24+ international and Chinese models (GPT/Claude/Gemini/Qwen/GLM/DeepSeek/Kimi etc.) behind a single API key and a single Base URL, with pass-through pricing.

Since Khoj already documents the OpenAI-proxy path, the pattern I'd highlight is narrow: pointing that existing base URL field at a multi-model endpoint makes one AI Model API entry carry the whole spectrum — flagship, budget, and bilingual models behind a single credential, so the per-agent chat-model assignments and the Default/Summarizer split become pure model-name choices instead of account juggling. The same works with LiteLLM, OneAPI, or any equivalent; the recipe is the point, not the product.

What I'd like to contribute

A quality-per-tier benchmark for Khoj's intermediate steps. I'd run the invisible workload — intent detection, query reformulation for web search, passage summarization, research-task synthesis — across budget, mid-tier, and flagship models through one endpoint, measuring task quality (does the intent classification match a frontier-model baseline? does the summary preserve the facts the downstream answer needed?), latency, and cost. My hypothesis is that intent detection and summarization are close to free-lunch territory for cheap models while final research synthesis is not — but the docs deserve numbers, not hypotheses. The output is a data-backed "choosing your Default and Summarizer models" section, vendor-neutral and usable regardless of which endpoint anyone runs. Given that automations fire unattended on a schedule, even a 5x cost difference on intermediate steps compounds silently — this is the benchmark your automation users need and nobody has published.

A "one AI Model API, many chat models" recipe extending the existing use-openai-proxy docs — provider-neutral prose covering: pointing the base URL at any multi-model OpenAI-compatible endpoint, which model-type and vision flags to set for gateway-fronted models, honest guidance on the tokenizer and max-prompt-size fields for common model families, and a recommended split (budget model for Default/Summarizer, flagship for advanced chat, bilingual model for non-English knowledge bases). Written to work identically with LiteLLM, OneAPI, or any equivalent.

A question about dynamic model discovery. Would a contribution that auto-populates Chat Model entries by fetching the model list from an OpenAI-compatible endpoint be welcome? The offline/Ollama path already hints at this direction (the docs mention restarting the server to load models available on the local provider, with manual entry as the fallback) — extending that discovery to any OpenAI-compatible base URL would remove the manual-row-per-model workflow entirely and feed the model pickers in the agent and user settings with what the endpoint actually offers. I'd be happy to spec the approach (which fields could be inferred, which should stay manual) and prototype it if maintainers think it fits the roadmap.

To be transparent: AnyLLM is my product and I benefit from awareness — but the benchmark and recipe are endpoint-agnostic, and the auto-discovery contribution helps every OpenAI-compatible provider equally. If contributions without product mentions fit the project's norms better, that works too.

Would the intermediate-step benchmark be a good starting point? Happy to share raw numbers from the intent-detection and summarization passes first.

Leo Bennett [email protected]