Decision memo: one LLM platform for twelve teams

This is a decision memo of the kind a staff engineer writes before an organisation commits to shared infrastructure. The organisation is hypothetical and every number is a stated assumption, but the structure, decisions and trade-offs are the real ones. Copy the shape; replace the numbers with yours.

• Twelve teams each integrated one or more model providers directly. Assume 14 separate API keys, three providers, and five different retry and timeout implementations.
• Spend is visible only as one monthly invoice per provider. No team can state its own cost per query; finance cannot allocate the bill.
• Model changes ship without a shared eval bar. A provider deprecation notice requires 12 teams to find and change their own code.
• Two teams send support transcripts containing personal data; neither has redaction before the call, and the data-processing terms differ by provider.
• Assume combined spend of $150k/month, growing 10% per month. That growth, not today's bill, is the reason to decide now.

Stand up one platform owned by a three-person platform team: an LLM gateway that every model call goes through, plus an eval service that gates model and prompt changes. Buy or adopt the gateway proxy; build the policy, budget and eval layers on top. Migrate all 12 teams over two quarters, highest-spend first. Teams keep ownership of their prompts, evals and product behaviour; the platform owns access, routing, budgets, policy and the gate.

• Tenant = team × environment. Each gets its own virtual key; provider keys never leave the gateway. Revoking a team is one action.
• Every request carries team, use case and data class (public / internal / confidential / regulated). Requests without them are rejected: the gateway fails closed.
• Isolation from noisy neighbours: per-tenant rate limits in requests and tokens per minute, carved out of each provider's org-level limit, with 20% reserved for customer-facing traffic. Batch and eval traffic goes to the batch API or a separate low-priority queue.
• Logs are tenant-scoped. Prompt and response logging is on by default with a 30-day TTL, off for the regulated data class, and readable only by the owning team and the platform on-call.

Showback for one quarter, then chargeback. Every request is costed at the gateway from token counts and the price table, including cache reads, cache writes and batch discounts, and attributed to team and use case. Each team gets a monthly budget with alerts at 50%, 80% and 100%; at 120% non-customer-facing traffic is throttled, never customer-facing traffic.

Be honest in the memo about where savings come from, because they are the case finance will check. Each lever is measurable at the gateway: shared system prompts moved to cached prefixes (cache reads at 0.1× input price), offline work moved to batch APIs (about 50% off), and routing simple intents to a smaller model behind an eval gate. If the levers cannot be shown to reach the break-even line within two quarters, the justification has to be risk and velocity (deprecations, personal data, incident response), and the memo should say so plainly.

• Allowlist by data class. Regulated data only to providers under a signed data-processing agreement in an approved region; confidential data never to providers that train on inputs.
• Three tiers, named by capability not vendor: small (classification, extraction), standard (most generation), frontier (hard reasoning, agents). Teams request a tier; the platform maps tiers to models.
• Pinned model versions, never floating aliases, for customer-facing traffic. Deprecations become a platform ticket with the eval run attached, not 12 surprises.
• Fallback per tier to a second provider, pre-evaluated. Fallback is triggered on provider errors and timeouts, never silently on quality.
• Prompts are versioned artefacts stored with the team's code, referenced by version in each request so every logged response maps to an exact prompt.

No model, tier mapping or customer-facing prompt change reaches production without passing its team's gate. The platform provides the harness, judge models, result storage and the CI check; teams provide the golden sets and thresholds. Minimum bar: 100 or more cases per customer-facing use case, the pass threshold agreed with the product owner, judge prompts validated against human labels before they gate anything, and a Canary & ramped rollout at 5% with automatic rollback on error-rate or cost regression.

Buy or adopt the commodity: the proxy that speaks every provider's API, retries, streams and counts tokens is a solved problem and changes monthly as providers add features. Build what is specific to us: the data-class policy, the chargeback model, the eval gate wired into our CI, and the dashboards finance reads. Revisit at 12 months: if the adopted proxy is more than 30% of platform on-call load, evaluate a managed gateway.

• Pilot with one customer-facing and one internal team, so both latency and policy are tested early.
• Migration is an SDK swap plus metadata; the platform team pairs with each team for the first day.
• Gateway overhead budget: under 20 ms p95 added latency; measured from day one and published.

• Positive: one place to see cost per team and per use case; a deprecation is one change; personal data policy is enforced, not requested; incidents have one log to search.
• Negative: a new critical dependency; three engineers who are not building product; every team pays a one-time migration cost of roughly a week.
• Reversibility: high for the proxy (swappable behind our SDK); medium for chargeback (finance will build on it); low for data-class policy once regulators or customers rely on it, which is why it is decided carefully now.
• Revisit trigger: if steady volume on any single open-weights model crosses the self-hosting break-even, open the self-host memo.