Vol V · Section 2
ML infrastructure & serving
Parallelism, GPU scheduling, multi-LoRA serving, autoscaling and prefix-aware routing.
1Explain data, tensor and pipeline parallelism and FSDP at a whiteboard level. How would you lay out training of a 70B model on 1,024 H100s?2How would you schedule a shared GPU fleet across training and inference?3How would you serve 2,000 fine-tuned customer adapters on top of one base model?4How should LLM inference autoscale? Why not on GPU utilisation, and what do you use instead?5What is prefix-aware routing for LLM serving, and when is it worth building?6Design checkpointing and preemption handling for large training jobs.7Design the data platform behind fine-tuning and RL: SFT sets, preference pairs and RL rollouts for many teams.8How would you roll out a new model version to production using shadow traffic and canaries?9How would you attribute LLM and GPU costs to teams on a shared platform?10Design multi-region LLM inference with data-residency guarantees across the EU, US, India, Singapore and the UAE.