How would you design a multi-tenant RAG system?
- 3 min — requirements: tenant count, size distribution, compliance tiers.
- 5 min — isolation models and the tiered mapping.
- 5 min — enforcement: fail-closed scoped client, quotas, noisy-neighbour controls.
- 4 min — capacity and cost per tier.
- 3 min — per-tenant eval, cost attribution, migration between tiers.
Pick an isolation model and know its trade-offs. Shared index with tenant filter: cheapest, scales to many small tenants, highest leakage risk, noisy-neighbour issues. Namespace/partition per tenant: good middle ground, physical separation within one cluster. Dedicated index or cluster per tenant: strongest isolation and the only thing that satisfies some compliance regimes, expensive and operationally heavy at scale.
Tiered approach: small tenants on a shared index with mandatory partition keys, enterprise tenants on dedicated namespaces, regulated tenants on dedicated clusters with their own encryption keys and region. Make tier a configuration, not a fork of the code.
The design constraint that matters: tenant scoping must be impossible to forget. Not a parameter callers pass, but a tenant-scoped client obtained from request context, where an unscoped query fails closed at the data layer. Also plan for per-tenant quotas (one tenant's bulk ingest must not starve others), per-tenant cost attribution, and per-tenant eval sets — quality varies enormously by corpus and an aggregate number hides a badly served customer.
Org scope: the platform team owns the tenant-scoped data client and the tier config; product teams can't bypass it. Buy the vector store (managed, with namespace support) and build the tenancy layer — that layer is where your isolation guarantee lives, so it should be your code with your tests.
Assumptions: 1,000 small tenants × 10k chunks; 50 enterprise × 400k chunks; 3 regulated × 2M chunks. 768-dim int8 vectors (768 B) + HNSW links (~128 B) + metadata (~100 B) ≈ 1 KB/chunk in RAM.
- Shared tier: 10M chunks ≈ 10 GB. Enterprise namespaces: 20M ≈ 20 GB. Each regulated cluster: 2 GB (tiny, but dedicated).
- Shared + enterprise fit on one 3-node replicated cluster of 64 GB nodes: 3 × ~$0.50/h × 730h ≈ $1.1k/month → ~$1.10/month per small tenant, ~$9/month per enterprise namespace (pro-rata by memory).
- Regulated: minimum 3 nodes for HA even at 2 GB → ~$1.1k/month floor per tenant, plus KMS and in-region logging. Price that tier at ≥$3–5k/month or it loses money.
- QPS: 200 peak across all tenants; ~2ms CPU per query → 0.4 cores. Memory and isolation, not CPU, drive the cost. (Node prices are illustrative on-demand memory-optimised, Oct 2026.)
What the answer sounds like at each level
Store a tenant id on every chunk and filter by it on every query.
Compares shared, namespace and dedicated isolation, tiers tenants by size and compliance, and adds per-tenant quotas and cost attribution.
Makes scoping impossible to forget via a fail-closed tenant-scoped client, prices tiers from the per-tenant cost floor, and owns per-tenant quality eval so no customer hides behind an average.
Follow-ups the interviewer will push
1One enterprise tenant bulk-ingests 20M chunks overnight. What protects everyone else?
Per-tenant ingestion quotas and a separate queue per tier, embedding concurrency caps per tenant, and index writes throttled so query latency on shared nodes stays within SLO. Large tenants move to their own namespace or nodes before the ingest, triggered by a size threshold.
2A regulated healthcare tenant requires EU data residency and its own keys. What changes?
Dedicated cluster in an EU region, customer-managed KMS keys, embeddings and LLM calls routed to EU endpoints, logs and traces stored in-region with PII redaction. Same code path; the tier config selects the cluster, region and key.
3The provider for your managed vector store has a regional outage. What happens?
Shared-tier tenants fail over to a replica region if residency allows; regulated tenants degrade to a 'search unavailable' answer rather than cross-region failover that would breach residency. Decide this per tier in advance and write it into contracts.
Red flags interviewers listen for
- Relies on callers remembering to pass a tenant filter.
- One isolation model for all tenants regardless of size or compliance.
- No per-tenant quotas, cost attribution or quality measurement.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Isolation design | Tiers isolation by tenant size and compliance with explicit trade-offs. |
| Enforcement | Scoping enforced at the data layer and fails closed. |
| Economics | Computes cost per tenant per tier and ties pricing to it. |
| Operations | Quotas, per-tenant eval and a path to move tenants between tiers. |
Sources
- pgvector: open-source vector similarity search for Postgres — pgvector, 2021docs
- Text Embeddings Reveal (Almost) As Much As Text — Morris et al., 2023paper