Noisy neighbour
One tenant's heavy load (a bulk import, a runaway agent) degrading latency or quota for everyone sharing the same infrastructure.
A noisy neighbour is a tenant whose heavy usage slows down everyone else on shared infrastructure. One customer starts a bulk import of 2 million documents or an agent stuck in a loop, and suddenly every other customer's searches are slow or their requests hit rate limits. The fix is to give each tenant fair limits and to keep any single tenant from consuming the whole shared pool.
One flat in the building running every tap at once, so everyone else's shower drops to a trickle.
Rate limiting only per user or per API key. One tenant with 500 users or 20 keys sails past every limit and still starves the shared provider quota.
LLM systems share several scarce pools: provider rate limits (tokens/minute per API key), GPU KV-cache and batch slots, vector-DB CPU, and ingestion workers. Protect each with per-tenant quotas (token buckets in the LLM gateway keyed by tenant), weighted fair queuing so a tenant with 10,000 queued jobs doesn't block one with 3, and separate pools for batch vs interactive. AWS's Builders' Library describes the approach: quota per tenant, admission control, and shuffle sharding, where each tenant is assigned a small random subset of workers so a bad tenant affects only tenants sharing that exact subset: with 8 workers and shards of 2 there are 28 combinations, so any one other tenant shares both workers with probability 1/28. Agents make neighbours noisier: a loop can burn hundreds of thousands of tokens in minutes, so cap per-run tokens and steps (Stop conditions & step budgets). Measure per-tenant consumption on every request so you can see who is noisy before you guess.
Per-tenant token buckets (tokens/min and requests/min) at the LLM gateway; separate queues and worker pools for batch/ingestion vs interactive; weighted fair queuing across tenants; per-run token and step caps for agents; shuffle sharding for stateful workers; per-tenant metrics on every request.
A developer-docs SaaS shares one provider key (2M input tokens/minute) across 400 tenants. One tenant launches a CI job that calls the assistant on every pull request: 900 requests/minute at 8k tokens = 7.2M tokens/minute demanded. Without quotas, everyone gets 429s. With a per-tenant bucket of 150k tokens/minute (burst 300k) and a separate batch queue, the CI tenant is throttled to its quota and queued; other tenants' p95 stays at 2.8 s.
I'd meter every request by tenant, enforce per-tenant token buckets at the gateway, separate batch from interactive pools, and use shuffle sharding where workers are stateful. Follow-up: 'what about your biggest customer who legitimately needs more?' A higher, contracted quota and possibly dedicated capacity, priced; fairness doesn't mean equal.
Numbers worth knowing
- Shuffle-shard overlap, 8 workers, shard size 2
- 1 in 28chance another given tenant shares both of your workers
- Per-tenant quota
- ~5–10% of shared poolfor SMB tiers; contracted higher for enterprise
- Runaway agent burn
- 100k+ tokens/mina looping agent with a 30k-token context calling every few seconds
How to decide
| When | Do this |
|---|---|
| Shared provider key across tenants | Per-tenant token buckets at the gateway. |
| Bulk ingestion and live queries share workers | Separate pools; batch gets leftovers. |
| One tenant regularly >20% of capacity | Dedicated pool or key, priced into their contract. |
| Agents run autonomously | Per-run token, step and time caps. |
Cost maths
Unmetered, the noisy tenant pays nothing extra while 399 tenants' requests fail. With metering: the CI tenant's 7.2M tokens/min × 60 min at $3/M = ~$1,300 per hour of demand. Quotas plus usage-based overage turn that into revenue or a conversation, instead of an outage.
Operating it
- Any tenant >20% of a shared pool
- Provider 429s that correlate with one tenant's activity
- Enterprise tenants with SLAs sharing pools with free-tier traffic
- Top tenant's share of tokens/min: alert >20% for 10 min
- Per-tenant 429 rate: alert when tenants other than the top one see >1%
- Agent runs exceeding token cap: alert per occurrence for review
When it goes wrong
Every tenant of an e-commerce catalog-enrichment product saw 429 errors for 50 minutes on a Monday morning.
Provider 429 rate went from 0 to 60%; per-tenant metrics (added afterwards) would have shown one tenant at 85% of tokens.
A large merchant started a 400k-SKU re-enrichment through the interactive API path; rate limits existed only per API key, and the merchant used 12 keys.
Per-tenant quotas at the gateway, batch API path with its own queue, and per-tenant token dashboards.
Where it shows up
Sources
- Fairness in multi-tenant systems — AWS Builders' Library, 2021blog
- Workload isolation using shuffle-sharding — AWS Builders' Library, 2019blog
Check yourself
Three questions. Every option is explained once you answer.