LLM gateway
A proxy between your apps and model providers that centralises auth, routing, fallback, rate limits, budgets, caching, redaction and logging.
An LLM gateway is a single service that every application calls instead of calling model providers directly. It holds the API keys, routes each request to the right model or provider, retries or falls back when a provider fails, enforces per-team budgets and limits, and logs every call in one place.
Without one, each team re-implements retries, key management and cost tracking, and nobody can answer 'how much did we spend on which model last week?'
A company's switchboard and expense desk rolled into one: every outside call goes through it, it knows who is allowed to call whom, and it sends the bill to the right department.
Putting prompts, retrieval and business rules into the gateway because it is convenient. It becomes a shared monolith every team must change through, and a bad deploy takes down every AI feature at once.
Core responsibilities: identity and keys (virtual keys per team or tenant; provider keys never leave the gateway); routing (by model alias, so support-default can move from one model snapshot to another without app changes); resilience (Model fallback & routing, retries with Exponential backoff and jitter, circuit breakers, timeouts per route); quota (Rate limiting and spend budgets per key, so one team cannot starve another, the Noisy neighbour problem); data controls (PII redaction, allow-listed models per data classification, regional routing); observability (tokens, cost, latency, cache hits and model per request, tagged with team and prompt version); and optional response or semantic caching.
Options: LiteLLM proxy (open source, OpenAI-compatible API over 100+ providers), Portkey, Envoy AI Gateway, Kong AI Gateway, Cloudflare AI Gateway, or the cloud providers' own. The gateway is now on the critical path, so it needs the same SLOs as your API tier: run ≥2 replicas per region, keep added latency to single-digit milliseconds, and avoid making it a buffering point that breaks streaming. Do not let it become a second application: business logic, prompts and evals belong to the apps; the gateway enforces policy.
LiteLLM proxy (or Portkey / Envoy AI Gateway) as a stateless deployment, ≥2 replicas per region, Redis for rate-limit and budget counters, Postgres for keys and spend; model aliases per use case; virtual keys per team with monthly budgets and per-minute token limits; per-route timeouts; streaming passthrough; OpenTelemetry export with team, alias, model, prompt version and cost on every span.
Five product teams, three providers. Before: 11 provider keys in 7 repos, no per-team cost view. After: LiteLLM proxy, 2 replicas per region; virtual keys per team with monthly budgets ($8k support, $3k internal tools…); alias docs-qa → primary model, fallback to the same model on a second cloud provider, then a smaller model; Presidio redaction for the HR and claims routes; OpenTelemetry spans to the tracing backend. First month: a runaway batch job from the internal-tools team hits its budget and is cut off at $3k instead of the $19k it was on track to spend.
Say: "A gateway gives me one place for keys, budgets, routing, fallback, redaction and cost attribution; it enforces policy but holds no product logic." Expect: "isn't it a single point of failure?": replicate per region, keep it stateless with limits in Redis, and allow a break-glass direct path with alerting.
Numbers worth knowing
- Added latency budget
- ≤10 ms p99 in-regionexcluding redaction; measure with streaming on
- Replicas
- ≥2 per regionit is on the critical path of every AI feature
- Budget enforcement granularity
- per virtual key, per minute and per monthminute limits stop runaways; monthly limits stop slow leaks
How to decide
| When | Do this |
|---|---|
| More than two teams or two providers | Introduce a gateway with virtual keys and aliases before keys spread further. |
| Single team, single provider, early prototype | Use the provider SDK directly with its built-in retries; add the gateway when a second consumer appears. |
| Regulated data on some routes | Enforce redaction and model allow-lists per route at the gateway, with audit logs. |
Cost maths
Gateway cost: 2 regions × 2 replicas × 1 vCPU / 2 GB ≈ $150–250/month plus Redis and Postgres ≈ $100–200. Benefit: per-team budgets and cache visibility. Example: routing the shared 6k-token system prompt through provider prompt caching (reads at 0.1× input, writes at 1.25× for the 5-minute TTL) at 400k requests/day at $3 / M input saves 400k × 6k × ($3 − $0.30) / M ≈ $6,480/day, visible only because the gateway attributes cache hits.
Operating it
- More than ~1,000 requests/s per region: horizontally scale and move counters to a Redis cluster.
- A tenant needs data residency: add region-pinned routes and per-tenant policy.
- Provider outages hurt more than once a quarter: add cross-provider fallback for critical aliases.
- Gateway-added latency p99 per route: alert above 25 ms.
- Spend per virtual key vs budget: warn at 80%, block at 100%.
- Error rate split by gateway vs provider origin: page when gateway-originated errors exceed 0.5%.
When it goes wrong
Every AI feature in a company returned errors for 40 minutes, including ones on healthy providers.
Synthetic checks on three products failed simultaneously; the gateway's own health check stayed green.
A gateway config deploy added a new route with a malformed regex in the redaction policy; every request paid a catastrophic-backtracking regex, saturated CPU and timed out. The health endpoint did not exercise the request path.
Config changes now go through a canary replica with real-path synthetic requests; regexes are compiled and fuzzed in CI; health checks send a real redaction-and-route request.
Related terms
Where it shows up
Sources
- LiteLLM documentation — BerriAI, 2026docs
- Prompt caching — Anthropic, 2026docs
- Rate limits — Anthropic, 2026docs
Check yourself
Three questions. Every option is explained once you answer.