Vol IV · Section 1
Evaluation
Golden sets, rubrics, LLM-as-judge, human review and the statistics behind a real improvement.
1You have a new LLM feature and zero eval data. How do you build a golden set from scratch?2What is the difference between offline and online evaluation, and why do you need both?3How do you write a good rubric for grading open-ended LLM output?4What are the known biases of LLM-as-judge, and how do you control for them?5When would you use pairwise comparison instead of absolute scoring?6How do you run human annotation for evals, and how do you know the labels are trustworthy?7Your new prompt scores 2 points higher on the eval. How do you know it is a real improvement?8When is synthetic data useful for evals, and what are its risks?9What is benchmark contamination, and how does it affect model selection?10What does eval-driven development look like day to day?