How do planning and reflection improve an agent, and when are they wasted effort?
- 2 min — planning and reflection defined; Reflexion and evaluator-optimizer
- 3 min — grounded vs ungrounded reflection
- 3 min — migration example with ablation numbers
- 2 min — revisable plans and caps
- 2 min — scaffold library and re-ablation at model upgrades
Planning makes the model write an explicit decomposition before acting, then execute and revise it; a visible plan is also a contract a human or a validator can check. Reflection has the model, or a second call, critique an output or trajectory and retry — Reflexion (Shinn et al., 2023) showed large gains on coding tasks when the critique was grounded in test feedback. Anthropic's evaluator-optimizer pattern (2024) is the workflow form: generator plus critic with explicit criteria and an iteration cap.
With reasoning models (extended thinking), much of this happens inside one call, so explicit scaffolding adds less than it did in 2023. The rule: reflection is worth its tokens only when there is an external signal to reflect on — failing tests, a schema validator, a compiler, a rubric judge calibrated against humans. Ungrounded self-critique tends to leave answers unchanged or talk the model out of correct ones.
SDK migration agent in a 400-file TypeScript service: “move from v3 to v4 of the payments SDK”. Plan step (one call, ~2k output tokens): list call sites via code search, group by API (14 groups), order by dependency, define done as tsc clean + unit tests green. Execution updates one group per sub-task. Reflection is grounded: after each group, run tsc and the affected tests; failures go back to the model with the compiler output. On a 40-repo eval, grounded reflection lifted success from 58% to 77% at +35% tokens; an ungrounded “review your diff” pass added +20% tokens and +1 point.
Treat each scaffold as an ablation on the eval set: success gain vs added p95 latency and cost per successful task, measured per model generation — a scaffold that paid off on a 2024 model may be dead weight on a reasoning model. Plans must be revisable: trigger re-planning on observations (a tool result invalidates step 3), not on a fixed schedule, and cap re-plans (2–3) so the agent doesn't plan forever. Use a separate judge with a rubric rather than self-grading when no programmatic signal exists, and calibrate it on ~100 human-labelled examples.
Org scope. The platform team owns the scaffold library (plan tool, evaluator-optimizer primitive, re-plan triggers) and an ablation harness; product teams turn scaffolds on per use case only with ablation results attached. Re-run ablations at every model upgrade — this is where most dead token spend hides. Failure mode: a rigid plan followed after step 2 has been proven wrong, or reflection loops that consume budget without changing the answer.
| Variant (n=40 repos) | Success | Tokens | p95 |
|---|---|---|---|
| Act only | 52% | 1.0× | 6 min |
| + plan | 58% | 1.1× | 7 min |
| + plan + self-review | 59% | 1.3× | 9 min |
| + plan + tsc/test feedback | 77% | 1.45× | 10 min |
What the answer sounds like at each level
Planning breaks the task into steps first; reflection lets the model check its work and try again.
Grounds reflection in tests, validators or a calibrated judge; makes plans revisable on observations; measures scaffolds as ablations with cost and latency.
Owns a scaffold library and ablation harness, re-evaluates scaffolds at each model upgrade, and removes ones reasoning models have made redundant.
Follow-ups the interviewer will push
1Your new reasoning model scores the same with and without your planning step. Keep it?
Keep the plan only if it has another job — human review, progress UI, checkpointing for durable execution. Otherwise remove it; it is pure latency and tokens.
2How many reflection rounds?
Cap at 2–3 and stop early when the external signal passes. Gains after round 2 are usually small and the risk of regressing a correct answer rises; measure the per-round delta on the eval set.
3Now the task has no programmatic check (e.g. a legal memo). What do you reflect on?
A rubric judge, separate from the generator, calibrated against lawyer labels (target ≥0.7 agreement), with criteria as checkable items: cites the governing clause, flags jurisdiction, states assumptions.
Red flags interviewers listen for
- Adds a “critique your answer” step with no external signal and no ablation.
- Treats the initial plan as fixed regardless of tool results.
- Ignores that reasoning models already plan internally.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Grounding | Ties reflection to tests, validators or a calibrated judge. |
| Measurement | Ablation with success, p95 and cost per successful task. |
| Currency | Accounts for reasoning models reducing the value of explicit scaffolds. |
| Org scope | Central scaffold library; re-ablation at every model upgrade. |
Sources
- Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al., 2023paper
- Building effective agents — Anthropic (Schluntz & Zhang), 2024blog
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper