How do you approach red teaming and jailbreak defence for an LLM product?
- 2 min — threat categories and the lethal trifecta
- 4 min — building and automating the attack suite
- 4 min — layered defences and blast-radius limits
- 3 min — metrics, fail-closed choices, ownership
Red teaming is structured adversarial testing to find ways a system can be made to behave harmfully: jailbreaks that bypass safety behaviour, prompt injection through retrieved documents or tool output, data exfiltration, and tool misuse. The highest-risk shape is Willison's lethal trifecta — access to private data, exposure to untrusted content, and a way to send data out — in one agent. No single filter or system prompt is sufficient; defence is layered, and the goal is limiting blast radius when an attack succeeds.
Before launching an agent that reads claim files and drafts emails for an insurance claims team: a red-team set of ~600 attacks across 8 categories (direct jailbreaks, role-play, encoded instructions, injection in PDFs, injection in email replies, cross-tenant reads, exfiltration via links or tool arguments, destructive tool calls), part hand-written, part generated by an attacker model in the style of Perez et al. Every successful attack becomes a regression case in CI. Defences: input and output guardrail classifiers, least-privilege tool scopes tied to the end user's permissions, human approval before any outbound email, and retrieved content treated as data, never instructions.
Assume jailbreaks will sometimes succeed and design for it: the model never holds credentials beyond the current user's, outbound channels are allowlisted, and irreversible actions need a human. Track attack success rate (ASR) per category per release — targets like <2% for direct jailbreaks and 0 successful irreversible actions (enforced architecturally, not by the model) — and track false-refusal rate on ~300 benign-but-borderline prompts with an alert above 3%, or the product becomes useless. Guardrail classifiers on the tool-call path must fail closed when they time out; budget them at ≤50 ms p95. Ownership: a central security/AI-safety team owns the attack library and the methodology and runs quarterly human red-team exercises; feature teams own fixing findings within an SLA, and the attack suite runs in every team's CI.
Attack suite: 600 cases × 2 repeats × (2k in + 300 out generation + ~$0.003 classifier) ≈ 1,200 × $0.0135 ≈ $16 per CI run. Quarterly human red team: 4 people × 3 days ≈ 12 person-days.
What the answer sounds like at each level
Test the system with malicious prompts and add filters to block jailbreaks.
Build a categorised attack suite, automate it in CI, layer guardrails, least privilege and human approval, and track attack success and false-refusal rates.
I design for successful attacks — break the lethal trifecta architecturally, enforce limits outside the model — and run a central attack library and quarterly exercises with fix SLAs across teams.
Follow-ups the interviewer will push
1The guardrail classifier service is down. What happens?
Chat-only responses can degrade to a stricter static policy; anything on the tool-call path fails closed — no outbound actions until the classifier is back. That choice is made in design review, not during the incident.
2A customer uploads a PDF with hidden instructions to email its contents to an outside address. Where does the defence live?
Not in the prompt. The email tool only allows recipients in the claim's contact list, every outbound email needs human approval, and the injection becomes a CI regression case.
Red flags interviewers listen for
- Relies on a system-prompt instruction like 'never reveal secrets' as the defence.
- Runs red teaming once before launch and never automates it.
- Doesn't measure false refusals.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Threat model | Covers direct jailbreak, indirect injection, exfiltration and tool misuse; names the trifecta. |
| Architecture | Enforces limits outside the model: scopes, allowlists, approvals, fail-closed tool path. |
| Measurement | Tracks ASR per category and false-refusal rate with thresholds, in CI. |
Sources
- Red Teaming Language Models with Language Models — Perez et al., 2022paper
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned — Ganguli et al., 2022paper
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Greshake et al., 2023paper
- The lethal trifecta for AI agents — Simon Willison, 2025blog