Chain-of-thought
Having a model write intermediate reasoning steps before its final answer, which improves accuracy on multi-step problems at a token cost.
Chain-of-thought means asking the model to 'think step by step', writing out its working before giving an answer. For maths, logic and multi-step questions, this leads to more correct answers.
It helps because the model produces one Token at a time; writing intermediate steps gives it room to work things out instead of jumping straight to a guess. Reasoning models & extended thinking are trained to do this automatically.
Doing long division on paper instead of in your head. The paper does not make you smarter, but it lets you hold intermediate results instead of guessing the final number.
Treating the visible reasoning as a faithful explanation of the model's decision. It can be post-hoc or simply wrong while the answer is right, so it is not a reliable audit log.
Introduced with worked-reasoning exemplars (Wei et al., 2022): PaLM 540B on GSM8K went from 17.9% to 56.9%. Zero-shot 'Let's think step by step' (Kojima et al., 2022) gets much of the gain without examples, and self-consistency — sample several chains at temperature > 0 and majority-vote the answer — added +17.9 points on GSM8K (Wang et al., 2022). It works because each generated token is extra computation the next step can condition on.
Since late 2024 CoT is mostly trained in: Reasoning models & extended thinking learn long reasoning via RL with verifiable rewards, so prompting them to 'think step by step' adds little and can hurt. Two caveats matter in production. The written chain is not a faithful account of the computation: Turpin et al. (2023) showed models' explanations omit the real cause of their answer (such as a biased example ordering). And CoT costs output tokens and latency; on lookup, extraction and simple classification it adds cost without accuracy.
Non-reasoning models: ask for reasoning in a separate field (e.g. a reasoning key before answer in the JSON schema) only on routes where the eval shows a gain. Reasoning models: give goals and constraints, not step-by-step scripts, and control depth with the effort or budget setting. Never persist chains as explanations for regulated decisions.
Financial-filings QA: 'Operating income fell from $412M to $356M; what was the percentage decline?' A small model answering directly says 15.7% about one time in three (it divides by the new value). With CoT it writes '412 − 356 = 56; 56 / 412 = 0.136' and reaches 13.6% reliably, at ~60 extra output tokens ($0.0009 at an assumed $15/M output). Self-consistency with 5 samples raises cost 5× and is only worth it on the 10% of questions flagged multi-step.
State when it helps (multi-step arithmetic, logic, planning on non-reasoning models), when it does not (lookup, extraction, reasoning models), and the faithfulness caveat: 'I never store the chain as an audit trail; I store the evidence and the decision.' Expect: 'how would you get CoT accuracy without showing it to users?' — reason in a hidden field or use a reasoning model, return only the answer and citations.
Direct answer
- Q -> answer
- Fewest tokens
- Fails on multi-step
- Fast
Chain-of-thought
- Q -> steps -> answer
- +50-500 output tokens
- Better on multi-step
- Not a faithful log
Numbers worth knowing
- GSM8K, PaLM 540B
- 17.9% → 56.9%standard vs CoT prompting, Wei et al. 2022
- Self-consistency gain
- +17.9 points GSM8KWang et al. 2022, majority vote over sampled chains
- Extra output
- 50–500 tokens per answerprompted CoT on non-reasoning models
How to decide
| When | Do this |
|---|---|
| Multi-step arithmetic or logic on a non-reasoning model | Prompted CoT in a hidden field. |
| Lookup, extraction, simple classification | No CoT; it adds cost and latency. |
| You need an explanation for an auditor | Store retrieved evidence and structured decision features, not the chain. |
Cost maths
Extra cost = CoT tokens × output price × calls × samples. 1M filings questions/month × 200 CoT tokens × $15/M = $3,000; self-consistency with 5 samples = $15,000. Route: apply 5-sample voting only to the ~10% flagged multi-step → $3,000 + $1,200 extra = $4,200.
Operating it
- Multi-step error rate stays > 10% with prompted CoT → move those requests to a reasoning model
- CoT tokens > 3× answer tokens on a route with no measured gain → turn it off
- Regulators or customers ask for explanations → build evidence-based explanations
- CoT tokens per answer p95 by route
- Accuracy with vs without CoT on the golden set — revisit monthly; turn off if delta < 1 point
- Counterfactual consistency of explanations < 95% on audited samples
When it goes wrong
An insurance-claims triage system stored the model's chain-of-thought as the explanation for each decision; an internal audit found explanations that cited policy clauses the decision did not depend on.
Counterfactual testing: removing the cited clause from the input did not change the decision in 11% of sampled cases.
Chain-of-thought text is not a faithful record of how the answer was produced (Turpin et al., 2023).
Replaced free-text explanations with the retrieved policy passages plus structured decision fields, and added the counterfactual test to the release eval.
Related terms
Sources
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al., 2022paper
- Large Language Models are Zero-Shot Reasoners — Kojima et al., 2022paper
- Self-Consistency Improves Chain of Thought Reasoning in Language Models — Wang et al., 2022paper
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Turpin et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.