Design the data platform behind fine-tuning and RL: SFT sets, preference pairs and RL rollouts for many teams.
- 5 min — data assets and users
- 10 min — registry, lineage, PII and decontamination
- 10 min — labelling, synthetic data, feedback loop
- 10 min — RL rollouts, verifiers, reward hacking
- 5 min — estimate and ownership
Four data assets: SFT examples (prompt, ideal response); preference data (prompt, chosen, rejected) for DPO or reward models; RL environments with verifiers (unit tests, math checkers, rubric judges); and rollouts (trajectories with rewards and log-probs).
The platform is a versioned dataset registry with lineage (source, licence, consent, PII scan, decontamination against eval sets); a labelling pipeline mixing human annotation, synthetic generation and production feedback; and rollout infrastructure — a vLLM or SGLang generation fleet feeding the trainer, e.g. GRPO (DeepSeekMath, later DeepSeek-R1) with groups of 8–16 samples per prompt.
A team fine-tunes a tool-calling agent for insurance claims: pull consented, PII-redacted production traces; keep successful ones by trajectory eval; generate synthetic variants; human reviewers label 2k; register claims-agent-sft@v3 by content hash; decontaminate against claims-eval@v5 (13-gram overlap + embedding near-duplicates); train a LoRA SFT; then RL with verifiers (tool-call schema validity, outcome check against a sandboxed claims API) → eval gate → model registry with lineage back to the datasets.
- Lineage is the legal moat. Every example traces to a source with rights. When a user exercises GDPR erasure, you must find their examples and the models trained on them in hours, not weeks.
- Reward hacking: verifiers that check only format get gamed. Review top-reward trajectories, keep holdout verifiers, add a KL penalty.
- Failure modes: contaminated evals inflating scores; model-collapse from training on its own outputs (keep a floor of real and human data); duplicates; noisy preference labels (rater agreement below κ ≈ 0.6 — see agreement).
- Org scope: a data-platform team owns registry, lineage and pipelines; each team owns its datasets and verifiers; privacy and legal own policy. Buy labelling capacity; build the registry and rollout infrastructure on open frameworks (veRL, OpenRLHF, TRL).
Assumptions: GRPO on an 8B policy; 512 prompts × 16 samples = 8,192 rollouts per step at ~4k tokens; generation on 32 H100s at ~5k tok/s each; train step + weight sync ~60 s; 1,000 steps; illustrative $2.50/H100-hour.
- Tokens per step: 8,192 × 4k ≈ 33M; generation ≈ 33M / 160k ≈ 205 s; with training ≈ 4.5 min/step → ~75 h for 1,000 steps.
- GPU-hours: (32 rollout + 16 trainer) × 75 h ≈ 3,600 ≈ $9k per run. Generation is ~75% of wall clock, so rollout throughput (async rollouts, smaller groups, shorter max length) is the lever.
- Data volume: ~150–200 GB of trajectories, rewards and log-probs per run; keep full data 90 days, samples indefinitely.
- SFT side: 10 teams × 50k examples × 2k tokens ≈ 1B tokens — a few GPU-hours per LoRA epoch. Labelling dominates: 2k human-labelled examples × ~$3 ≈ $6k per dataset version.
What the answer sounds like at each level
Stores JSONL training files in a bucket and runs fine-tuning scripts.
Versions datasets with lineage and decontamination, and runs rollouts on an inference engine for RL.
Designs lineage for erasure and licensing, treats verifiers as attack surfaces, sizes the rollout fleet as the bottleneck, and sets team ownership boundaries.
Follow-ups the interviewer will push
1A user requests deletion under GDPR Article 17.
Lineage lookup finds their examples; remove them from datasets, flag affected model versions, and record the retrain-or-assess decision. The platform's job is to make the lookup possible in hours.
2RL reward climbs but evals stay flat.
Reward hacking. Inspect the top-reward trajectories, harden the verifier, add a holdout verifier and KL penalty, and gate on the held-out eval, not reward.
310 teams become 50.
Self-serve dataset registration, per-team rollout GPU quotas, a shared verifier library, and standard eval gates.
Red flags interviewers listen for
- No lineage from model back to data source.
- No decontamination against eval sets.
- Reward is trusted without inspecting trajectories.
- Ignores that rollout generation dominates RL wall clock.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Governance | Lineage supports erasure and licensing; consent and PII enforced at ingest. |
| Quality | Decontamination, dedupe, agreement-checked labels, real-data floor. |
| RL infrastructure | Rollouts on vLLM/SGLang sized as the bottleneck; verifier hardening. |
| Org | Clear split between platform, dataset owners and legal. |
Sources
- Training language models to follow instructions with human feedback — Ouyang et al., 2022paper
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al., 2023paper
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Shao et al., 2024paper