How should LLM inference autoscale? Why not on GPU utilisation, and what do you use instead?
- 5 min — why GPU utilisation fails; what saturates in an LLM engine
- 10 min — signals, thresholds, hysteresis
- 10 min — cold start, warm pools, predictive minimums
- 10 min — admission control, degradation, failure modes
- 5 min — savings estimate and ownership
nvidia-smi 'utilisation' reports the share of time any kernel is running, so it reads near 100% at almost any load — useless as a load signal. LLM engines saturate on other resources:
- Queue depth — requests waiting for a batch slot (
vllm:num_requests_waiting). - KV-cache usage — vLLM exposes a KV-usage gauge; above ~90%, preemption is imminent.
- SLO signals — TTFT and inter-token p95.
Scale out on the leading indicators (queue, KV) and confirm with the lagging one (TTFT p95). Cold start is minutes, so headroom and predictive minimums matter as much as the trigger.
A KEDA ScaledObject on Prometheus: average waiting per replica > 3 for 60 s, or KV usage > 85%, → add replicas. Scale in only after waiting = 0 and KV < 50% for 15 min, at most −10% per step (hysteresis). A warm pool of provisioned nodes with weights on local NVMe cuts cold start from ~6 min to ~90 s. A scheduled minimum replica count follows the same hour over the last four weeks.
- Failure modes: thrash from short windows; a scale-out storm pulling 70 GB of weights from object storage at once and saturating bandwidth (use a regional cache or peer-to-peer distribution); long-context traffic inflating KV usage with no queue (give it its own pool).
- When scaling can't keep up: admission control — return 429 with Retry-After to the batch tier once the queue passes a threshold, protect interactive traffic, degrade to a smaller model.
- Fail static: if the metric pipeline dies, hold current replicas and alert rather than scaling to zero.
- Org scope: the serving platform owns the policy; product teams own SLO tiers. KEDA or HPA with custom metrics is enough; managed endpoints (SageMaker, Vertex, Bedrock provisioned throughput) do it for a premium.
Assumptions: 70B FP8 on 4×H100 replicas; a replica sustains ~12 QPS at TTFT p95 ≤ 1 s; daily peak 400 QPS, trough 80 QPS; cold start = node provisioning 3–5 min + 70 GB weight pull at ~1.5 GB/s (~47 s) + engine init and CUDA graphs ~45 s; illustrative $2.50/H100-hour.
- Replicas at a 70% target: peak 400 / 8.4 ≈ 48 replicas (192 GPUs); trough 80 / 8.4 ≈ 10 (40 GPUs).
- $: static-for-peak 192 × 730 × $2.50 ≈ $350k/month; autoscaled average ~25 replicas (100 GPUs) ≈ $183k/month → ~$170k/month saved — only if released GPUs are on-demand or lent to batch.
- Headroom: traffic ramps ~7 QPS/min at the morning peak. A 6-min cold start needs ~40 QPS ≈ 5 replicas of headroom; a 90-s warm start needs ~1.
- Thresholds: waiting > 3 per replica for 60 s or KV > 85% → scale out; 15 min calm → scale in, max −10% per step.
What the answer sounds like at each level
Autoscales on GPU utilisation or request rate with an HPA.
Uses queue depth and KV usage as triggers, TTFT as confirmation, and adds hysteresis and warm pools.
Sizes headroom from cold-start time and ramp rate, designs admission control and fail-static behaviour, and quantifies savings only where GPUs are actually released.
Follow-ups the interviewer will push
1Traffic jumps 10× in two minutes after a push notification.
Autoscaling cannot react in two minutes: pre-scale on the marketing calendar, queue with admission control, degrade to a smaller model, and keep prefix caches warm.
2The region is out of GPUs.
Scale-out fails; route to another region if residency allows, or to an API provider through the gateway. Alert on pending pods > 0 for 5 min.
3Which signal for a prefill-heavy RAG workload?
TTFT p95 and queue depth; KV usage stays low because outputs are short. Tokens-in-flight is a better capacity proxy than request count.
Red flags interviewers listen for
- Autoscales on GPU utilisation.
- Ignores a multi-minute cold start.
- Symmetric scale-in and scale-out with no hysteresis.
- No admission control when scaling can't keep up.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Signals | Queue depth and KV usage as leading, TTFT as lagging; rejects GPU utilisation with a reason. |
| Dynamics | Headroom derived from cold start × ramp rate; hysteresis on scale-in. |
| Overload | Admission control, tiered shedding, fail-static scaler. |
| Economics | Savings counted only when GPUs are released or reused. |
Replacing GPU-utilisation autoscaling with queue and KV signals
Illustrative exampleOur serving platform ran ~25 internal applications on ~300 H100s. Autoscaling keyed on GPU utilisation kept the fleet pinned at maximum replicas around the clock, yet we still breached the TTFT p95 SLO about 9 times a month at peak.
I was tech lead of the four-person serving team and owned fixing both the cost and the SLO breaches without new hardware.
I pulled 30 days of engine metrics and showed utilisation sat at 97–100% at both 50 and 400 QPS, while the waiting-queue gauge rose ~90 s before the TTFT breach in 11 of 12 incidents. I proposed KEDA on waiting-per-replica plus KV usage, with a warm node pool. The SRE lead objected that custom metrics put Prometheus in the scaling path. We agreed a fail-static rule — if the metric is missing for 2 minutes the scaler holds replicas and pages — plus an HA Prometheus pair. I wrote the scaler config and a load-test harness; two engineers built the warm pool with weights on local NVMe. We rolled out to 2 apps, then 10, then all 25 over five weeks.
Over the next 60 days SLO breaches fell from ~9 to 1 per month, average GPUs in use dropped from ~300 to ~210, and the freed capacity ran batch evals (~65k GPU-hours/month). Cold start went from 6.5 min to 80 s. In hindsight I would have split long-context traffic out first: three long-context tenants triggered needless KV-driven scale-outs in week one.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper
- KEDA documentation — KEDA project (CNCF), 2025docs
- vLLM documentation — vLLM project, 2025docs