Discussion: a hybrid posture for the two-model split — local completion, optional cloud chat tier

Hi Tabby team,

I've been following Tabby for a while — a Rust code-completion server where ten developers share one RTX 3090 instead of each needing a Copilot subscription is a genuinely different economic proposition, and the Answer Engine with its visible source trace is a standout feature.

I wanted to open a discussion about a pattern I keep hitting with Tabby deployments, and a possible way to serve it.

The two workloads inside Tabby have different constraints

Tabby's architecture already separates two model roles: a completion model and a chat model. What strikes me is how different their real-world constraints are:

Completion is latency-bound and privacy-critical. Ghost text needs sub-200ms first-token latency to not break flow, and it's constantly streaming raw code context. This tier belongs on-prem — local models on a shared GPU are exactly right, and Tabby's serving stack (Tree Sitter prompts, adaptive caching) is purpose-built for it. Chat and the Answer Engine are quality-bound. When a developer asks "why does this service keep deadlocking" or the Answer Engine synthesizes across an indexed repo, they'll wait several seconds — but they expect frontier-grade reasoning. This is where self-hosted deployments quietly suffer: the same 1.5B-7B chat model that's fast enough for completion is often the weak link for the tasks where quality actually matters. Right now teams face a binary: run everything local and accept a chat-quality ceiling, or route the whole thing to a cloud API and give up the privacy story that made them choose Tabby in the first place.

The middle path: route only the chat tier, keep completion on-prem

I've noticed Tabby has been adding hosted model options recently — Avian as a provider, Mistral's embedding API. So the "local-only" posture is already softening where it makes sense. I'd suggest the natural next step is making the hybrid posture first-class: completion stays local on the GPU box, chat/Answer Engine optionally points at an OpenAI-compatible endpoint, and the privacy boundary is explicit — raw code context never leaves, only user-typed questions and the snippets the developer chooses to include.

This is where I think my project fits. I'm the developer of AnyLLM, a lightweight OpenAI-compatible API gateway that fronts 24+ models from both Western and Chinese labs behind one API key and one base URL, pass-through pricing, international card / crypto payment. Concretely, for Tabby this would mean:

A docs recipe for the hybrid deployment. Tabby's chat model config already accepts an OpenAI-compatible endpoint, so pointing it at a gateway is a pure config change — no code, no protocol work. One recipe covering "local completion + cloud chat" would formalize a pattern I suspect many teams are improvising anyway.

A quality-per-tier benchmark across the chat model ladder. Something like: local 1.5B → local 7B → DeepSeek-V3-class → frontier models, all answering the same set of repo-grounded questions through the Answer Engine. If local models hold up on most queries, that's a great story for Tabby's local tier — and knowing exactly where the ceiling sits helps teams decide what, if anything, to route out. I'd happily run this and publish the methodology and results either way they land.

Access to the Chinese coding model lineage. Tabby's recommended models already lean heavily on Chinese open-weights — DeepSeek-Coder, Qwen, StarCoder lineage. The same labs now ship API-only frontier coding models (Qwen3-Coder, GLM-4.x, Kimi K2) that are strong at repo-scale reasoning and aggressively priced, but genuinely hard to access from outside China: separate accounts, international card requirements, regional restrictions. A gateway with pass-through pricing and international payment support is the bridge.

To be clear, this isn't a request to change Tabby's local-first direction — that direction is correct and is Tabby's whole identity. The suggestion is narrower: document and support the escalation path for the one tier where local quality genuinely caps out today, while leaving completion exactly where it is.

Happy to contribute the docs recipe, the benchmark harness, or a working test deployment — whichever is most useful. And I'd genuinely like to hear the team's read on whether the "local completion feels great, chat feels capped" pattern matches what you hear from self-hosted teams.

Thanks for the project — the end-to-end philosophy (IDE extensions and serving optimized as one stack, not a thin wrapper) shows in the latency numbers.

Leo Bennett [email protected]