Thinking models, explained without the hype

Most AI chat products now offer two kinds of model, or one model with a switch. One answers straight away. The other pauses, sometimes shows a line like “thinking…”, and then answers, usually better on hard problems and noticeably slower. The second kind is called a reasoning model or “thinking model”. This post explains what it is actually doing during that pause, what you pay for it, and how to decide which one to use.

A large language model writes its answer one small piece of text at a time. Those pieces are tokens, roughly three-quarters of a word each in English. Each new token is chosen by looking at everything written so far, which is what next-token prediction means. There is no separate “thinking” step hidden inside a normal model: the writing is the computation.

That has a surprising consequence. If a model has to produce the final answer to a hard maths problem as its very first word, it gets one shot at it. If it is allowed to write out intermediate steps first, every step it writes becomes something the next step can look at. Writing is how it gets more room to work.

Researchers noticed in 2022 that simply asking a model to “think step by step” made it better at maths and logic puzzles. That technique is called chain-of-thought prompting. Reasoning models take the idea further: they are trained, mostly with reinforcement learning on problems that have checkable answers, to produce long internal working before they answer. The DeepSeek-R1 paper (2025) describes this kind of training openly, and reports the model learning on its own to re-check and backtrack.

So when a thinking model pauses, it is writing. It may write several thousand tokens of working: trying an approach, noticing a mistake, trying again. Many products hide that working or show only a short summary, but it is still generated, and it is still billed.

Extra working helps most when the task has several dependent steps and a way to be checked: a maths word problem, a bug that needs tracing through three files, a schedule with ten constraints, a contract clause that must be compared against two others. In those cases a mistake early on poisons everything after it, and the chance to notice and correct it is exactly what the working provides.

It helps much less when the answer is a fact the model either knows or does not (“what is the capital of Peru?”), or when the task is about tone rather than logic (“make this email friendlier”). Thinking harder does not create knowledge the model never learned, so it is not a cure for made-up facts. A thinking model can reason very carefully from a wrong premise.

Both costs come from the same place: those extra tokens. Providers charge per token, and the hidden working is charged like the visible answer, at the output price, which is usually several times the input price. Speed works the same way: a model produces tokens at a roughly steady rate, so every token of working is time before you see your first word of the answer. That delay is what engineers call time to first token.

Here is a worked example. The prices and speed are assumptions chosen to make the arithmetic easy, not any provider's real rates: input at $3 per million tokens, output at $15 per million, and the model producing 100 tokens per second.

On those numbers the thinking answer costs eleven times as much and takes forty seconds longer. For one question from one person, that is nothing. For a customer-support product answering a million questions a month, it is the difference between $6,000 and $66,000, which is why teams care about cost per query.

The curve flattens for a reason. Research on spending more computation at answer time (Snell et al., 2024) found that the best amount depends on how hard the question is: easy questions gain little, and very hard ones can stay out of reach however long the model works. Many products now let you choose an effort level, low to high, precisely because no single setting suits every question.

• Use a thinking model for problems with several steps that depend on each other: debugging, maths, planning a project, comparing detailed documents, checking an argument for holes.
• Use a standard model for quick facts, rewriting, summarising, translation, brainstorming and chat, where speed matters and there is no long chain of logic to get wrong.
• Use a low effort setting when you are unsure. Raise it only if the answer was wrong in a way that more care would have fixed, such as an arithmetic slip or a missed condition.
• Do not use thinking as a fact-checker. If the question depends on recent or specialised facts, the fix is giving the model the source material (that is what retrieval does), not more thinking.