When is synthetic data useful for evals, and what are its risks?
- 2 min — what synthetic data is good for
- 4 min — generation recipe for retrieval and adversarial slices
- 4 min — realism gap, filtering, separate reporting
- 2 min — hand-over to real traffic
Synthetic eval data is generated by an LLM: questions generated from chunks of your corpus for retrieval evals (the RAGAS approach), paraphrases of existing cases, adversarial variants, unanswerable questions for abstention, and simulated users for multi-turn agents. It is cheap — dollars per thousand items — and covers situations real traffic rarely shows. It supplements a golden set built from real users; it never replaces it.
Internal wiki assistant, 5,000 documents, no users yet: sample 2,000 chunks, generate one question per chunk with the source chunk recorded, and you have a retrieval eval that measures recall@10 on day one. Then generate targeted slices production would take months to produce: multi-document questions, questions whose answer was deleted from the wiki (must abstain), and 200 jailbreak and injection attempts embedded in page content.
Synthetic questions are easier and more literal than real ones: they reuse the chunk's own wording, which flatters lexical retrieval and overstates recall. Counter it by prompting with 20 real query exemplars for style, generating from a document summary rather than the chunk, and measuring the gap — if BM25 beats dense retrieval on synthetic items but loses on real ones, the synthetic set is lexically leaky. Filter hard: dedupe at embedding similarity > 0.9, check answerability with a second model, human spot-check 100 items (expect to drop 30–40%). Use a different model family to generate than to grade, or they share blind spots. Tag every item source: synthetic and report its slices separately; once real traffic exists, real items become the gate and synthetic items move to coverage slices.
2,000 chunks × (1.5k in + 200 out) at $3/$15 per M = 3M in ($9) + 0.4M out ($6) ≈ $15. Answerability check with a second model ≈ another $10. Filtering keeps ~1,300 items; human spot-check of 100 at 3 min ≈ 5 h.
What the answer sounds like at each level
You can use an LLM to generate test questions when you don't have real data, but they might be too easy.
Generate per-chunk questions for retrieval evals and targeted adversarial and abstention slices; filter, dedupe, spot-check, and report synthetic separately from real.
I treat synthetic data as a coverage tool with a measured realism gap, use a different model family to generate than to judge, and have a policy for when real traffic takes over the gate.
Follow-ups the interviewer will push
1How would you detect that your synthetic set is too easy?
Compare the same systems on synthetic and on a small real slice: if scores are 10+ points higher on synthetic, or lexical retrieval wins only on synthetic, it is leaky. Measure query-chunk token overlap too — real queries share far fewer exact terms.
2Can you fine-tune on the same synthetic data you evaluate on?
Only with a strict split by source document; otherwise you train on paraphrases of your test items and the eval measures memorisation. Keep eval-generation prompts and seeds separate from training-data generation.
Red flags interviewers listen for
- Uses the same model to generate items and to judge answers.
- Mixes synthetic and real items in one score with no tagging.
- Ships synthetic items with no answerability filter or human spot-check.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Use-case fit | Uses synthetic for coverage and cold start, not as the sole gate. |
| Bias awareness | Explains lexical leakage and proposes a measurable check. |
| Hygiene | Dedupe, answerability filter, human sample, cross-family generator/grader, tagging. |
Sources
- RAGAS: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023paper