Design multi-region LLM inference with data-residency guarantees across the EU, US, India, Singapore and the UAE.
- 5 min — scope: regions, tenant tiers, legal requirements
- 10 min — control plane vs regional data planes; routing
- 10 min — every data path: logs, caches, traces, evals, backups
- 10 min — failover, degradation, capacity and cost
- 5 min — ownership and verification
Residency means data — prompts, outputs, logs, traces, KV caches, embeddings, eval samples, backups — is stored and processed inside a declared boundary. Architecture: a global control plane (config, model registry, metadata with no content) and regional data planes (gateway, inference pools, stores, logging, keys). Routing follows the tenant's residency policy, not the user's IP: an EU tenant's traffic stays in the EU even when the user travels. Failover stays inside the boundary; single-region jurisdictions may have no failover, only degradation.
Tenant config: residency: eu, allowed_regions: [eu-central-1, eu-west-1]. Geo-DNS sends a travelling user to the nearest edge; because TLS termination is processing, the edge passes the connection through at L4 (or terminates only in-boundary) to the EU gateway. Logs stay in-region, encrypted with per-region KMS keys. Support staff elsewhere get just-in-time, audited, redacted access. Model weights replicate everywhere — they contain no customer data.
- Legal map: GDPR (EU; transfers outside the EEA need adequacy or SCCs, and many customers contract for EU-only regardless of the 2023 EU-US Data Privacy Framework), India's DPDP Act 2023, the UAE PDPL, Singapore's PDPA. Legal owns the policy matrix; sales cannot promise regions outside it.
- Cost: each boundary needs its own peak plus in-boundary failover headroom, so residency costs ~1.3–2× a global pool. GPU supply differs by region, which may force smaller models in some — publish per-region model availability.
- Leak paths: traces exported to a US observability vendor, support tooling, caches, eval pipelines sampling production, default cross-region backups. Test with tagged canary records and egress monitoring.
- Org scope: the platform owns identical regional stamps via infrastructure-as-code; privacy owns policy. Use providers' regional endpoints where self-hosting isn't justified.
Assumptions: regional peaks EU 900, US 1,200, India 450, Singapore 300, UAE 150 QPS (sum 3,000); time zones make the simultaneous global peak ~2,250; 400 output tokens/request; ~1,275 output tok/s per H100 at SLO; illustrative $2.50/H100-hour.
- Per-region peak GPUs (QPS × 400 / 1,275): EU 282, US 376, India 141, Singapore 94, UAE 47 → 940.
- Residency-bound: +50% in-boundary headroom for failover and burst → 893 × 1.5 ≈ 1,340, plus the single-site UAE at 2× (94) → ≈ 1,434 GPUs.
- Global pool alternative: 2,250 × 400 / 1,275 ≈ 706 + 30% ≈ 918 GPUs.
- Residency premium: (1,434 − 918) × 730 × $2.50 ≈ $940k/month (~1.56×) — price it into enterprise residency tiers rather than absorbing it.
What the answer sounds like at each level
Deploys the model in several regions and routes users to the nearest one.
Routes by tenant policy, keeps logs and caches in-region, and fails over only within the boundary.
Prices the residency premium, maps every data path including telemetry and support, separates control and data planes, and assigns policy ownership to legal and privacy.
Follow-ups the interviewer will push
1The EU region loses capacity.
Fail over to the second EU region; if both are down, degrade in-boundary (smaller model, queue) rather than cross. Emergency cross-border processing only for tenants who have consented, as logged per-tenant config.
2A new country requires localisation and has no cloud GPU region.
Options: in-country colocation with owned GPUs (6–12 months, capex), a local cloud partner, or not offering it yet. Quantify demand before committing.
3Is the KV or prefix cache a residency issue?
KV is derived personal data in memory; it is fine inside in-region pools, but caches are never shared across boundaries, and provider prompt caching is used only through in-region endpoints.
Red flags interviewers listen for
- Routes by user IP rather than tenant policy.
- Fails over across the boundary by default.
- Forgets telemetry, traces, support tools and backups.
- Assumes residency is free.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Architecture | Global metadata-only control plane, regional data planes, tenant-policy routing. |
| Completeness | Enumerates non-obvious data paths and tests them with canary records. |
| Capacity | In-boundary headroom priced against a global pool. |
| Governance | Legal-owned policy matrix; sales bound to it. |
Sources
- Regulation (EU) 2016/679 — General Data Protection Regulation — European Union, 2016spec