Design an LLM-based content-moderation pipeline for a platform receiving 200M posts and comments per day.
- 5 min — scope: volume, surfaces, policies, regions, latency
- 10 min — cascade tiers and thresholds
- 10 min — humans, appeals, feedback loop
- 10 min — metrics, drift, adversaries, legal obligations
- 5 min — estimate and ownership
A cascade: cheap high-recall stages first, expensive precise stages on the uncertain residue.
- Tier 0 — hash matching against known-violating media (PDQ, PhotoDNA) and near-duplicate text.
- Tier 1 — small fine-tuned classifiers (DeBERTa-class, 100–400M parameters) per policy, with calibrated scores.
- Tier 2 — a policy-conditioned 8B safety model (Llama Guard-style) on items in the uncertain score band.
- Tier 3 — a frontier model with full policy text and context for hard cases and appeals.
- Humans on the residue and on appeals; their labels retrain tiers 1–2.
Thresholds are set per policy, per language, per surface.
Policies are versioned config: taxonomy, definitions, examples, thresholds and an action per severity per region (allow, label, downrank, hold, remove, escalate). High-risk surfaces (live comments on a minor's post) run tiers 0–1 synchronously before publish within a 300 ms budget; everything else is asynchronous. Child-safety matches bypass the LLM tiers entirely and go to a dedicated legal reporting path.
- Measure prevalence, not just accuracy. Precision and recall per policy per language from a stratified daily sample labelled by expert reviewers; top-line metric is violating views per 10k views; also appeal-overturn rate.
- Failure modes: over-removal of counter-speech and reclaimed terms; recall gaps outside English; adversarial drift (leetspeak, text in images, coded language) — monthly red-teaming and retraining; a policy edit silently shifting behaviour — policies are versioned and eval-gated like code.
- Reviewer welfare is a system requirement: exposure caps, blurring, rotation.
- Org scope: Trust & Safety policy owns definitions; the ML platform owns the pipeline; legal owns regional obligations (the EU Digital Services Act requires statements of reasons and transparency reporting). Build classifiers and the policy engine; buy hash-database access and, where justified, frontier API capacity.
Assumptions: 200M items/day, ~150 tokens each; tier 1 on 100%; tier 2 on 10% (500-token prompt, ~10 output tokens); tier 3 on 0.5% (1.5k in / 200 out at illustrative $3/M input, $15/M output); humans on 0.05%; illustrative $2.50/H100-h, $0.80/L4-h, $3k/month per reviewer seat.
- QPS: 200M / 86,400 ≈ 2,300 average, ~7k peak.
- Tier 1: ~1,500 items/s per L4 (DeBERTa-base, 256 tokens) → 5 L4 at peak, run 8 ≈ $4.7k/month.
- Tier 2: 700 items/s peak × 500 tokens = 350k prefill tok/s; an 8B model prefills ~20k tok/s per H100 → 18 GPUs, run 24 ≈ $44k/month.
- Tier 3: 1M/day × ($0.0045 + $0.003) = $7.5k/day ≈ $225k/month.
- Humans: 100k/day ÷ (150 decisions/h × 8 h) ≈ 85 reviewers ≈ $255k/month.
- Total ≈ $530k/month. Tier 3 and humans are ~90% of it, so each point of tier-2 precision is worth more than any GPU optimisation.
What the answer sounds like at each level
Sends every post to an LLM with the policy and acts on its answer.
Builds a cost-ordered cascade with calibrated thresholds, human review on the residue and a feedback loop.
Sets prevalence as the top metric, prices humans alongside GPUs, versions policy as config with eval gates, and splits ownership across policy, ML and legal.
Follow-ups the interviewer will push
1A new policy must launch in two weeks.
Ship it in the LLM tier first (policy prompt + ~200 examples), measured on a 2k labelled set. Collect LLM + human labels, distil a tier-1 classifier in 4–6 weeks, then restrict the LLM to the uncertain band.
2A spam wave brings 10× volume.
Tiers 0–1 scale horizontally and cheaply; add per-account rate limits. The LLM tier has a fixed budget: overflow on high-risk surfaces goes to 'hold' (fail closed); low-risk surfaces 'allow + async review'.
3An EU regulator asks why a post was removed.
Each decision logs policy version, tier, scores, model versions and reviewer id; the statement of reasons is generated from that record, not reconstructed.
Red flags interviewers listen for
- Every item goes to a frontier LLM — cost and latency are ignored.
- One global threshold for all policies and languages.
- No human review or appeal path.
- Child-safety content handled by the same generic LLM path.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Cascade design | Cost-ordered tiers with calibrated routing bands and fail-closed behaviour on high-risk surfaces. |
| Measurement | Per-policy, per-language precision/recall plus prevalence and appeal overturns. |
| Economics | Shows that the frontier tier and human review dominate cost. |
| Governance | Versioned policy-as-config, auditable decisions, regional legal duties. |
Sources
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations — Inan et al., 2023paper
- Moderation guide — OpenAI, 2025docs