How would you schedule a shared GPU fleet across training and inference?
- 5 min — workload shapes and SLOs
- 10 min — quota hierarchy, borrowing, priorities, preemption
- 10 min — gang and topology-aware placement, fragmentation
- 10 min — metrics and failure modes
- 5 min — governance and build vs buy
The workloads have different shapes. Inference: latency SLO, diurnal, many small (1–8 GPU) long-running services, needs fast scale-up. Training: throughput, gang-scheduled (all N GPUs or none), topology-sensitive, preemptible if it checkpoints. Batch (evals, embeddings, synthetic data): elastic filler.
Use a quota hierarchy with borrowing: guaranteed quotas per team, idle quota lendable, borrowed capacity preemptible by priority. On Kubernetes: Kueue (ClusterQueues, cohorts, borrowing and preemption) or Volcano; Slurm remains common for large training. Add topology-aware placement (same NVLink domain, same leaf switch).
2,048 H100s. Inference has a guaranteed 1,200 at peak but uses ~500 at trough, so ~700 GPUs are lendable for ~12 h/day to preemptible training and batch with a 2-minute drain contract (checkpoint on SIGTERM). Large runs book a contiguous 512-GPU block on one spine through a capacity calendar; small jobs queue. Priority classes: prod-inference > reserved research > preemptible batch.
- Allocated is not used. Track allocation rate and real use (SM activity, MFU) separately; allocated-but-idle GPUs are the largest waste. Also queue wait p90 per team, preemptions, and fragmentation (free GPUs that cannot form a whole node).
- Failure modes: 300 free GPUs scattered 1–3 per node; gang deadlock (two jobs each holding half of what they need); inference scale-up blocked by training that fails to drain; NCCL traffic contending on shared switches.
- Policy is political. Publish quotas, borrowing and preemption rules; review utilisation monthly. A GPU council (engineering leads + finance) allocates quota; the compute-platform team owns scheduler and policy. Adopt Kueue, Slurm or Run:ai; build the policy and reporting.
Assumptions: 2,048 H100s; inference needs 1,200 at peak and ~500 at trough, ~12 h/day near trough; training and batch absorb any lent capacity; illustrative $2.50/H100-hour.
- Static split (1,200 inference + 848 training): inference averages ~800 used → ~400 idle ≈ 400 × 730 × $2.50 ≈ $730k/month of idle reservation.
- With borrowing: lend 700 GPUs × 12 h × 30 days ≈ 252k GPU-hours/month ≈ $630k/month of recovered value.
- Preemption tax: two preemptions a day losing ~10 min each (half a checkpoint interval + restart) → 700 × 2 × 10/60 × 30 ≈ 7k GPU-hours ≈ 3% of borrowed hours.
- Fragmentation: whole-node (8-GPU) granularity for training, bin-pack inference onto dedicated nodes; target <5% stranded GPUs.
What the answer sounds like at each level
Splits the cluster into a training partition and an inference partition.
Uses quotas with borrowing, gang scheduling, priority preemption and checkpoint-on-SIGTERM.
Quantifies idle reservation, designs the borrowing contract and topology placement, separates allocation from utilisation, and sets up the governance that allocates quota.
Follow-ups the interviewer will push
1A team wants 1,024 GPUs for three weeks next month.
Book a contiguous topology block in the capacity calendar, shift batch to off-peak, possibly burst inference to an API provider; present the opportunity cost and get an explicit leadership decision.
2Inference spikes while training is borrowing capacity.
Preempt the lowest-priority borrowed gangs; they get SIGTERM and 120 s to checkpoint; inference pods start from nodes with weights pre-pulled. Target: capacity returned in ≤5 min.
3Allocation is 85% but average MFU is 20%.
Show per-job SM-activity dashboards, auto-reclaim idle notebooks after 2 h, right-size inference replicas, and report $ per useful FLOP to each team.
Red flags interviewers listen for
- Static partitions with no borrowing.
- No gang scheduling for multi-node jobs.
- Measures allocation and calls it utilisation.
- Preempts training with no checkpoint contract.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Scheduling model | Quotas with borrowing, gang scheduling, priority preemption with a drain contract. |
| Topology | Whole-node granularity, NVLink and spine locality for large jobs. |
| Economics | Quantifies idle reservation and recovered value. |
| Governance | Published policy, quota council, utilisation reviews. |
Sources
- Kueue documentation — Kubernetes SIG Scheduling, 2025docs
- The Llama 3 Herd of Models — Llama Team, Meta, 2024paper