Tell me about a time data changed your mind.
- 1 min — the belief and why you held it
- 2 min — the fair test and pre-registered rule
- 1 min — the result and how you communicated it
- 1 min — what changed, regret
This tests intellectual honesty: holding a strong position, setting up a fair test, and updating publicly when the data disagreed — including telling the people you'd argued with that they were right.
Good AI examples: believing fine-tuning would beat prompting, that a bigger model was needed, that chunk size didn't matter, or that users wanted longer answers.
Staff answers show the update changed a direction other teams depended on, and that you made the reversal easy for others to see and follow.
What the answer sounds like at each level
Changed their implementation approach after a benchmark contradicted their expectation.
Ran a fair test against their own proposal and reversed a team direction based on the result.
Reversed a cross-team investment publicly, credited the dissenter, redeployed budget, and installed pre-registered decision rules.
Follow-ups the interviewer will push
1How did you make sure the test was fair to fine-tuning?
The other engineer reviewed the fine-tuning setup, we used the same base model, and I tuned hyperparameters on a held-out split the eval never saw.
2What would have made you keep fine-tuning?
If the combined system had beaten retrieval alone by more than 2 points on jurisdictions with thin corpora — it did on one, by 3 points, so we kept a small fine-tune on the roadmap for that jurisdiction only.
Red flags interviewers listen for
- The 'data' was ambiguous and the candidate didn't actually change their mind.
- No credit to the person who held the right view.
- Test designed to confirm the original position.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Honesty | Clear reversal, stated publicly, credit to the dissenter. |
| Rigour | Pre-registered decision rule, expert labels, n stated. |
| Impact | Budget redeployed and production result measured. |
Fine-tuning lost to retrieval
Illustrative exampleI was convinced we needed to fine-tune an open-weights model for our legal-research assistant to get jurisdiction-specific answers right. I'd argued for it in planning; a senior engineer on another team argued that better retrieval over the statute corpus would get us further for less.
I was leading the effort with two engineers and had a quarter of GPU budget allocated on my recommendation.
Before spending the budget, I set up a fair comparison on 600 questions across six jurisdictions, labelled by two contract attorneys: a LoRA fine-tune, the base model with improved hybrid retrieval and a reranker, and both together. I pre-registered the decision rule: pick the cheapest option within 2 points of the best. Retrieval alone scored 81%, fine-tune alone 72%, both 82%. Retrieval won by the rule. I presented the result myself, credited the other engineer by name in the meeting, and returned 70% of the GPU budget.
We shipped the retrieval improvements in five weeks; jurisdiction-correct answers went from 64% to 80% in production sampling (n = 1,000). The returned GPU budget funded a red-teaming effort that found two prompt-injection paths before launch. My regret: I'd defended my position for a month in planning before testing it — a two-week experiment at the start would have saved both teams friction.