A model gateway is a service every model call in an organisation goes through, so that authentication, routing, quota, failover, logging and key custody happen once instead of in forty codebases.
It is a straightforward piece of engineering with two genuinely hard parts — correct streaming and distributed quota — and a maintenance burden that is larger than the build.
This page describes the architecture, and then states the costs.
What a gateway is for The case for one becomes obvious at a particular size.
Once several teams call models independently, five things are true at once: nobody can say what the organisation spends, provider keys are distributed across services and CI systems, switching providers means editing every repository, there is no consistent record of what was sent to a third party, and every team has written its own retry logic with different bugs.
A gateway addresses all five by being the single egress point.
That is its real definition: not a proxy, but the place where organisational policy about model use is expressed once and enforced.
It also introduces a hard dependency — a component that, when down, stops every AI feature in the company.
That is the central trade, and it means a gateway must be engineered to a higher availability standard than most of the services behind it.
Design for it to fail open where policy permits: a client library that can call a provider directly with a break-glass credential when the gateway is unreachable turns a total outage into a degraded mode, at the cost of a credential that must be audited.
The seven components Component Description Identity and authorisation Who is calling.
Per-service credentials, not one shared secret, so quota and audit can attribute anything.
Maps a caller to a set of permitted models and limits.
Router Turns a model alias into a concrete provider, model and endpoint, according to rules: cost, latency, capability, data residency, current health.
The component that makes provider choice a configuration change.
Quota and rate limiting Enforces per-caller limits on requests, tokens and spend, across all gateway replicas.
Distributed state, on the hot path, with a strict latency budget.
Provider adapters Translate a canonical request shape to each provider's API and back, including error mapping, token accounting and streaming semantics.
The bulk of the ongoing maintenance lives here.
Resilience Timeouts, retries with jitter and a bounded budget, circuit breakers per provider, and failover to the next route.
Implemented once, correctly, rather than in every client.
Audit and telemetry A durable record per request: caller, model, token counts, cost, latency, status, and — subject to policy — the content.
This is the artefact that answers compliance questions and produces the cost report.
Key custody Provider credentials held in one place, never distributed to callers, rotated on a schedule without a fleet-wide deploy.
The request path through them, in order: authenticate, resolve the alias, check quota, select a route, attach the provider credential, call, stream the response back while counting tokens, record the audit entry, decrement quota with actual usage.
Everything on that path is latency the caller pays, so the target is a few milliseconds of gateway overhead excluding the provider call — which mostly means the quota check must be fast and the audit write must be asynchronous.
Routing and the model alias The most valuable single feature is indirection: callers ask for a capability, not a vendor’s product name.
Three properties follow.
A provider migration is a pull request rather than a coordinated change across every team.
A canary is a weight change, which makes canary deploys for model changes possible without touching any caller.
And residency becomes enforceable centrally rather than by convention — a route that does not satisfy the constraint is not selectable, which is a much stronger guarantee than a policy document.
Keep the routing