Tell me about building consensus on an eval standard across teams.
- 1 min — fragmented metrics, the customer symptom
- 2 min — calibration data and private sharing
- 1 min — the dissenting team and the tiered compromise
- 1 min — adoption, the bug it surfaced, regret
Standards stick when teams help write them, when they cost little to adopt, and when they're enforced by tooling rather than goodwill. The interviewer wants to see you navigate genuine disagreement about what 'good' means.
Common cross-team eval debates: which faithfulness definition, whether LLM judges count as ground truth, how big a golden set must be, whether to gate merges on evals.
Staff answers end with a standard encoded in a platform — an eval harness and gate — adopted by most teams, with a documented exception process.
What the answer sounds like at each level
Agreed a metric definition within their own team.
Aligned several teams on a metric with calibration data and a shared tool.
Created an org-wide standard with co-authors, tooling enforcement, cost tiers for dissenters, and a governance model.
Follow-ups the interviewer will push
1Why calibrate against humans at all?
Because each team's metric had looked fine on its own; only a shared human-labelled set showed which metrics tracked reality.
2How do you keep the standard alive?
Owned by the eval platform, recalibrated each quarter and on every judge upgrade, with the labelling guidelines versioned alongside.
Red flags interviewers listen for
- Imposes the standard top-down.
- Treats an LLM judge as ground truth without calibration.
- Ignores adoption cost for the dissenting team.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Evidence | Human-labelled calibration with agreement statistics. |
| Consensus | Co-authored, dissent addressed with a cost tier. |
| Durability | Encoded in tooling with an owner. |
One faithfulness standard for four teams
Illustrative exampleFour teams shipping RAG features over financial filings each reported 'faithfulness' with different definitions — two with LLM judges, one with string overlap, one with weekly manual spot checks. Leadership saw four green dashboards while customers reported unsupported numbers in answers.
As the staff engineer for applied AI, I needed one standard the teams would actually use, with no authority over three of them.
I ran a calibration exercise: 250 answers sampled from all four products, labelled by two analysts for claim-level support. Inter-annotator agreement was κ 0.81. Against those labels, the four team metrics ranged from κ 0.32 to 0.71. I shared the results with each lead privately first. The team with the overlap metric objected that judges were too expensive at their volume. I proposed tiers: a claim-level judge on a 2% sample online plus a full run on a 400-item golden set per release, which cost them ~$90 a week. I wrote the standard with the four leads as co-authors and built the harness integration; my team of three maintained it.
All four teams adopted it within two months; the judge reached κ 0.76 against analyst labels. The standard surfaced a numeric-hallucination bug in one product that had been reporting 97% faithful — real rate 88% — fixed within a sprint. My regret: I didn't include the analysts in the standard's governance, and when their labelling guidelines changed six months later, the calibration set drifted without anyone noticing.