Design an LLM gateway for a company with many teams calling several model providers.
- 3 min — scope and capacity estimate: QPS, streams, nodes, spend
- 5 min — hot path vs async path; latency budget; SLO
- 4 min — trade-offs: unified vs pass-through, fail-open vs fail-closed
- 4 min — failure modes: 429 storms, bad deploys, buffering, ledger drift
- 3 min — org: team size, chargeback, build vs buy, rollout to 40 teams
An LLM gateway is the single internal service every application calls instead of calling providers directly. Scope for this answer: 40 teams, 3 hosted providers plus 1 self-hosted model pool. The hot path is deliberately thin: authenticate the virtual key → apply policy (allowed models, region, PII mode) → TPM token bucket that reserves max_tokens up front and reconciles to actual usage on completion → route (primary, fallback, self-hosted) → stream pass-through. Everything else is async: usage events → Kafka → cost ledger, redacted traces to the trace store, and an eval sampler feeding drift monitoring. Budget: p99 added latency under 15 ms excluding provider time, and a 99.95% availability SLO.
Each team gets scoped virtual keys tagged team / feature / tenant, so cost per query lands in the ledger by feature. The gateway enforces per-team TPM and daily budgets, retries transient errors with full-jitter backoff under a retry budget, opens a circuit breaker per provider-region, and fails over only to routes with their own eval results. It manages prompt caching headers consistently, because at this volume caching is the largest cost lever. Streaming responses pass through byte-for-byte with proxy buffering disabled.
Key trade-offs. Unified API vs pass-through: a lowest-common-denominator schema loses tool-use formats, cache controls and structured outputs, so offer the unified API as the default and a provider-native pass-through route with the same auth, quota and logging. Fail-open vs fail-closed: if the quota store is down, fail open (log and alert) — blocking every LLM call company-wide is worse than a short overspend; but for regulated tenants, PII redaction fails closed. Failure modes to design for: 429 storms that synchronised retries amplify; a bad gateway deploy taking down every product at once (canary by team with automatic rollback, plus an SDK break-glass flag that calls the provider directly with a sealed key); proxies or ingress buffering SSE streams and destroying TTFT; and chargeback drift where the ledger disagrees with the provider invoice — reconcile monthly and alert on >2% variance. Org: a 4–5 person platform team owns it, with showback first and chargeback once the numbers are trusted. Build vs buy: start from an open-source or managed gateway (LiteLLM, Envoy AI Gateway, Portkey, Kong as of Oct 2026) and build only the ledger, policy and routing logic specific to you — the proxy itself is not a differentiator.
Assume 5M calls/day, 80% streaming, 3k input / 400 output tokens, peak = 3× mean. QPS: 5M / 86,400 ≈ 58 mean, ~175 peak. Concurrency: 175 QPS × 8 s average stream ≈ 1.4k concurrent streams. Nodes: ~2k streams per node ceiling → 1.4k fits on one, so plan 3 (one per AZ, each ≤50% at peak) and run 6 (N+2 across 3 AZs, survives an AZ loss during a rolling deploy). Spend: 15B input + 2B output tokens/day at an assumed $3 / $15 per M (illustrative, Oct 2026) = $45k + $30k ≈ $75k/day. Prompt caching at a 60% hit rate on input, reads billed at 0.1×: saves 0.6 × $45k × 0.9 ≈ $24k/day before the 1.25× cache-write premium on misses. The gateway's own compute (6 mid-size nodes) is a rounding error next to that.
What the answer sounds like at each level
A central proxy that handles API keys, logs requests and retries failures, so teams don't each integrate every provider.
A thin hot path — auth, policy, token-bucket quota, routing, stream pass-through — with async usage and trace pipelines, a latency budget, circuit breakers and evaluated fallbacks.
I size it from traffic, set a 15 ms p99 overhead budget and a 99.95% SLO, design fail-open vs fail-closed per tenant class, reconcile the ledger to invoices, and run it as a 4–5 person platform product with chargeback and a buy-the-proxy, build-the-policy split.
Follow-ups the interviewer will push
1Now 10× traffic: 50M calls/day.
Mean ~580 QPS, peak ~1.75k, ~14k concurrent streams → ~7 nodes at 2k streams each; run 12 across 3 AZs. The gateway stays cheap; the real limits become provider TPM (spread across accounts and regions, add the self-hosted pool for bulk traffic), rate-limit state (shard Redis by key), and the Kafka usage pipeline (partition by team).
2A regulated healthcare tenant onboards.
Region-pinned routes, zero-data-retention endpoints only, PII redaction fail-closed inline, a separate encrypted trace store with 30-day TTL, a BAA with every provider on their routes, and their own quota pool so other teams can't starve them.
3The primary provider is down for 40 minutes.
The circuit breaker opens after error rate crosses 50% over 20 requests; interactive routes fail over to the evaluated second provider or the same model on another cloud; batch and eval traffic is paused to free capacity; product teams see a degraded-mode flag. Fallback rate is the headline metric on the incident dashboard.
Red flags interviewers listen for
- Unified API only, with no pass-through for provider-specific features.
- Buffers streaming responses or does synchronous logging on the hot path.
- Fails closed on quota-store outage for every tenant, or fails open on PII for regulated ones.
- No reconciliation between internal cost ledger and provider invoices.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Capacity math | Derives QPS, concurrent streams, node count with redundancy, and daily spend from stated assumptions. |
| Hot-path discipline | Keeps auth/policy/quota/route/stream inline with a numeric overhead budget; everything else async. |
| Failure design | Covers retry storms, gateway-as-blast-radius, stream buffering, and fail-open/closed per tenant class. |
| Org and economics | Sizes the team, plans showback→chargeback, reconciles to invoices, and buys the proxy layer. |
Sources
- LiteLLM Proxy documentation — BerriAI, 2025docs
- Prompt caching — Anthropic, 2025docs
- Site Reliability Engineering, ch. 21: Handling Overload — Google, 2016book