Backpressure
Pushing back on producers when a consumer is saturated: bounded queues, 429s and load shedding instead of unbounded buffering until latency or memory collapses.
Backpressure is how a busy part of a system tells upstream to slow down. Without it, requests pile up in a queue that grows forever: latency climbs, memory fills, and eventually everything times out at once. With it, the queue has a size limit, and when it is full the system rejects new work quickly (an HTTP 429 or 503) so callers can back off with Exponential backoff. LLM providers do this to you with rate-limit errors; your own services should do it too.
A restaurant with a full kitchen stops seating new tables and hands out a wait time, rather than seating everyone and serving nobody for two hours.
Treating an unbounded in-memory queue as 'buffering'. It converts a load spike into a latency spike plus an OOM, and every request in the backlog is already past its client timeout by the time it is served.
Queueing theory sets the shape: for an M/M/1 server, mean time in system is service_time / (1 − ρ), so latency doubles from 50% to 75% utilisation and is 10× at 90%. Above ~80% sustained utilisation, small bursts cause large tail latency. Backpressure caps that by bounding work in flight: a fixed-size queue or semaphore per dependency (Little's law: concurrency = throughput × latency, so 20 req/s at 6 s per LLM call needs 120 slots), and fast rejection when it is full. Reject early and cheaply: a 429 at the gateway costs microseconds; a request that queues for 40 s and then times out wastes the client's whole budget and your tokens. Prefer LIFO or deadline-aware dropping under overload (serve requests whose clients are still waiting), shed by priority (interactive over batch), and propagate deadlines so downstream stops working on requests the caller has abandoned. Kafka and SQS give natural backpressure for async work: consumers pull at their own rate and lag becomes the visible signal, but lag still needs an alert or it is just a slower outage.
Gateway token bucket per provider at 90% of the published TPM/RPM limit; bounded queue per dependency (size = Little's-law concurrency × 1.5); max queue wait 10 s for interactive, minutes for batch; 429 + Retry-After on overflow; priority classes interactive > batch > eval; deadlines propagated via request headers.
A developer-docs assistant calls a hosted model with a 400k tokens/minute limit. Average request is 6k tokens, so the ceiling is ~66 requests/min. The gateway holds a token-bucket per provider sized to 90% of the limit and a bounded queue of 40 requests with a 10 s max wait. During a launch-day spike of 200 requests/min, 70% get an immediate 429 with Retry-After: 8, the UI shows 'busy, retrying', and p95 for admitted requests stays at 9 s instead of every request timing out at 60 s.
Default: bound every queue and every concurrency pool, size them from Little's law, and reject at the edge with 429 plus Retry-After when full. I'd shed batch and eval traffic before interactive traffic. The follow-up is usually 'why not just autoscale?' Because GPU capacity takes minutes to arrive and provider rate limits don't scale at all; backpressure covers the gap autoscaling can't.
Numbers worth knowing
- Latency multiplier at 90% utilisation
- 10× service timeM/M/1 approximation; real tails are worse with bursty LLM traffic
- Concurrency needed
- throughput × latencyLittle's law: 20 req/s × 6 s = 120 in flight
- Cost of an edge 429
- <1 ms, 0 tokensversus a full model call that times out after 60 s
How to decide
| When | Do this |
|---|---|
| Interactive path hits a provider or GPU limit | Reject fast with 429 + Retry-After; let the client back off. |
| Async ingestion or eval batch | Use a pull-based queue (SQS, Kafka); let lag absorb bursts and alert on lag age. |
| Mixed traffic classes on one pool | Priority queues; shed batch first, never interactive first. |
| Overload persists >10 minutes | Step down the Graceful degradation & load shedding ladder (smaller model, cached answers) instead of only rejecting. |
Cost maths
Without backpressure, a timed-out request still pays for its tokens. Assume $3/M input and $15/M output (illustrative Oct 2026 list prices), 6k in + 500 out = $0.0255 per call. A 10-minute overload at 200 req/min where 70% of calls time out client-side after the model already ran wastes 1,400 × $0.0255 ≈ $36 of tokens and delivers nothing. Shedding them at the edge costs $0 and the clients retry into capacity.
Operating it
- Queue wait p95 >25% of the client timeout
- 429 rate >5% for more than 15 minutes: you need more capacity or a fallback, not a bigger queue
- Consumer lag age on the ingestion topic >30 minutes during normal load
- In-flight / max concurrency per dependency: alert >85% for 5 min
- Queue wait time p95: alert >25% of client timeout
- Edge 429 rate: alert >5% for 15 min
- Consumer lag age (Kafka/SQS ApproximateAgeOfOldestMessage): alert >15 min
When it goes wrong
A code-search assistant went from 4 s p95 to 100% timeouts during a company hackathon.
Synthetic probe failures and pod OOMKill events; the in-process queue held 9,000 requests.
An unbounded asyncio queue in front of a self-hosted vLLM pool. Traffic tripled, the queue grew, every queued request was already past the 30 s client timeout when served, and the pods ran out of memory.
Bounded semaphore of 96 per pool (from Little's law), 429 on overflow, deadline check before dispatch, and HPA on queue depth rather than CPU.
Related terms
Sources
- Site Reliability Engineering, ch. 21: Handling Overload — Google, 2016book
- Using load shedding to avoid overload — AWS Builders' Library, 2019blog
Check yourself
Three questions. Every option is explained once you answer.