Rate limiting
Capping requests and tokens per key, tenant or user per time window, usually with token buckets, so one caller cannot exhaust shared provider quota or budget.
Rate limiting caps how much any one caller can use per unit of time: requests per minute, input tokens per minute, output tokens per minute, or dollars per day. Model providers rate-limit you, and you should rate-limit your own users and teams so that one heavy user or runaway agent cannot use up the shared quota (the Noisy neighbour problem) or the budget.
When a limit is hit, the caller gets an error (HTTP 429) and should wait and retry, or the request is queued.
A water-park ride that lets 20 people on per minute: the queue still moves, but no single group can block it for an hour.
Limiting only requests per minute. One request with a 150k-token document costs as much quota as 1,000 short chat turns, so a request limit lets a single tenant exhaust token quota while staying under its request cap.
Providers limit on several axes at once. Anthropic, for example, enforces requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM) per model class with a token-bucket algorithm that refills continuously, so short bursts can trip a limit even under the per-minute average; for most current Claude models only uncached input tokens count toward ITPM, so Prompt caching raises effective throughput. A 429 carries a retry-after header; honour it rather than retrying immediately. A monthly spend cap returns a 429 without retry-after, and retrying will not help until it resets.
Your own limiter sits in the LLM gateway: token buckets per tenant and per user, keyed on estimated tokens (prompt tokens + max_tokens, reconciled after the response), stored in Redis with atomic Lua scripts. Allocate provider quota across tenants with weighted fair queuing or per-tenant ceilings that sum to ~80% of the provider limit, leaving headroom for retries. Agents need a per-task budget too (Agent cost & token budgets): a loop that calls the model 200 times is a rate-limit and cost problem simultaneously. Pair limits with Backpressure: queue low-priority work (batch, evals) and shed it first.
Token buckets in Redis (atomic Lua) at the gateway, keyed per tenant and per user, on both requests and estimated tokens; tenant ceilings summing to ≤80% of provider limits; separate queue for batch and eval traffic via the provider Batch API; honour retry-after, else exponential backoff with full jitter, max 3 retries; per-agent-task token and step budgets.
Legal-research SaaS with 40 tenants on one provider org with 2M ITPM. A single tenant's bulk 'summarise all 9,000 contracts' job sends ~1.8M input tokens/minute and every other tenant starts getting 429s. Fix: per-tenant token buckets with a ceiling of 15% of ITPM for interactive work and a separate batch queue drained by the provider's Batch API (50% cheaper, not counted against interactive limits). Interactive p95 latency returns to normal and the bulk job finishes overnight at half the price.
Say: "I rate-limit per tenant on tokens, not just requests, keep the sum of tenant ceilings under ~80% of provider quota, honour retry-after with jittered backoff, and route bulk work to a batch queue." Expect: "how do you estimate tokens before the response?": prompt tokens plus max_tokens, reconciled after.
Numbers worth knowing
- Anthropic Start tier, Sonnet-class (Oct 2026)
- 1,000 RPM, 2M ITPM, 400k OTPMper model class; higher tiers raise all three
- Tenant ceiling headroom
- sum of ceilings ≤ 80% of provider limitleaves room for retries and bursts
- Effective input throughput with caching
- 2M ITPM → ~10M total input tokens/min at 80% cache hitscache reads not counted toward ITPM on most current Claude models
How to decide
| When | Do this |
|---|---|
| Multi-tenant product on shared provider quota | Per-tenant token buckets plus weighted fair queuing; never first-come-first-served. |
| Work is not latency-sensitive (bulk summarisation, evals, backfills) | Send it through the Batch API or a low-priority queue, not the interactive limit. |
| 429s exceed 1% of interactive requests for a week | Request a higher tier, add caching for repeated prefixes, or spread across a second provider region. |
Cost maths
Rate limits are also cost limits. A runaway agent loop at 50 calls/min × 20k input tokens = 1M input tokens/min ≈ $3/min at $3 / M, $180/hour, $4,300/day if nobody notices. A per-task budget of 300k tokens caps it at ~$1. Batch API for bulk jobs halves the bill: 9,000 contracts × 12k tokens ≈ 108M input tokens ≈ $324 interactive vs $162 batched (input only).
Operating it
- Sustained use above 70% of a provider limit: request a tier increase before peaks hit it.
- One tenant consistently uses more than 30% of quota: give them a dedicated provider org or workspace.
- Retry traffic exceeds 5% of total: you are amplifying overload; tighten backoff and shed low-priority work.
- 429 rate by origin (provider vs own limiter) and by tenant: alert above 1% for interactive routes.
- Provider quota utilisation (RPM, ITPM, OTPM): warn at 70%, page at 90% sustained for 5 minutes.
- Top-5 keys by token share: alert when any exceeds its ceiling for 10 minutes.
When it goes wrong
An internal coding agent platform returned 429s to every team for two hours on a Monday morning.
Gateway 429 rate jumped from 0.1% to 38%; the spend dashboard showed one virtual key at 70% of org ITPM.
A scheduled agent job refactoring 400 repositories had no per-task step or token budget and retried 429s immediately without backoff, consuming the shared quota and amplifying the overload.
Per-task budgets (max 60 model calls, 500k tokens), per-key ceilings at 20% of org quota, retry-after-aware backoff with jitter, and batch jobs moved to a low-priority queue.
Related terms
Where it shows up
Sources
- Rate limits — Anthropic, 2026docs
- Timeouts, retries, and backoff with jitter — Brooker / Amazon Builders' Library, 2019blog
Check yourself
Three questions. Every option is explained once you answer.