In-context learning
A model adapting to a task purely from instructions and examples in its prompt, with no change to its weights.
In-context learning means a model picks up a new task from what you put in the Prompt & system prompt, without any retraining. Show it three examples of turning a product review into 'positive' or 'negative', and it will label the fourth.
Nothing about the model changes; it learns only for that one request, inside its Context window, and forgets when the request ends.
Handing a temp worker a sheet with three filled-in forms and saying 'do the rest like these'. They do not need training; they copy the pattern for this shift only.
Believing the model 'remembers' what it learned in a previous chat. Nothing is written to the weights; persistence needs Agent memory or retrieval.
ICL emerged with scale: GPT-3 (2020) showed few-shot performance rising sharply with model size. Mechanistic work identifies induction heads — attention circuits that find an earlier occurrence of the current pattern and copy what followed — as one driver (Olsson et al., 2022). Min et al. (2022) found that demonstrations teach the label space, input distribution and format more than the exact input→label mapping: randomising labels in demonstrations cost only a few points on many classification tasks.
ICL underpins Few-shot prompting, Retrieval-Augmented Generation (RAG) (facts in context) and agent tool use. Its costs and limits: every request pays for the examples in tokens and Time to first token (TTFT) (cut 90% of the input price with Prompt caching), recall degrades for information in the middle of long contexts (Lost in the middle), and results are sensitive to example order and phrasing. When the same examples ride on millions of calls, a fine-tuned small model is cheaper and more consistent.
Start every new task with instructions + 3–8 examples, the stable part placed first and marked for Prompt caching, the current date and variable data last. Measure on a 100–300 item Golden set before considering fine-tuning.
Insurance-claims routing: 12 labelled claim summaries (~1,500 tokens) plus the new claim reach 89% routing accuracy on day one with no training. At 2M calls/month those examples are 3B input tokens: $9,000/month at an assumed $3/M input, or ~$900 with cached reads at 0.1×. Six months later, a LoRA fine-tune of a small model on 5,000 routed claims reaches 93% without examples in the prompt.
Frame prompting, RAG and fine-tuning as rent vs buy: 'In-context learning rents a skill per request; fine-tuning buys it. I rent first, cache the prefix, and buy when volume and evals justify it.' Expect: 'does the model remember the examples next time?' — no; persistence needs Agent memory, retrieval or training.
In-context learning
- Examples in the prompt
- Weights unchanged
- Works in minutes
- Pays tokens every call
Fine-tuning
- Examples in training
- Weights updated
- Needs a training run
- No per-call example cost
Numbers worth knowing
- Cached input price
- 0.1× base inputAnthropic cache reads; writes 1.25× (5-min TTL) or 2× (1-hour)
- Label-randomisation penalty
- a few points on many tasksMin et al. 2022, classification
- Typical example budget
- 3–8 examples, 300–2,000 tokensclassification and extraction
How to decide
| When | Do this |
|---|---|
| New task, < 100k calls/month | In-context learning with prompt caching. |
| > 1M calls/month and a stable task | Fine-tune a smaller model and drop the examples. |
| The needed knowledge changes weekly | Keep it in context via Retrieval-Augmented Generation (RAG), never in weights. |
Cost maths
Monthly example cost = calls × example tokens × input price × (cache-hit share × 0.1 + miss share × 1.0) + cache writes. 2M calls × 1,500 tokens × $3/M = $9,000 uncached; at a 98% hit rate ≈ $9,000 × (0.98 × 0.1 + 0.02) = $1,062. Compare with fine-tuning: ~$100 of GPU plus ~$1,000/month to host a small model.
Operating it
- Example tokens > 50% of input tokens on a route with > 1M calls/month
- Accuracy gains flatten beyond 8 examples
- Cache hit rate < 80% because the prefix keeps changing
- Prompt-cache hit rate < 80% on high-volume routes
- Example tokens as share of input tokens > 50%
- Output PII detections of entities absent from the current input > 0
When it goes wrong
A claims assistant occasionally mentioned a policyholder's name that had nothing to do with the claim being processed.
An output PII scanner flagged names not present in the current claim or retrieved documents.
The few-shot examples were copied verbatim from real production claims; the model echoed details from the demonstrations into unrelated replies.
Replaced demonstrations with synthetic or fully redacted examples, added a PII check on prompt templates in CI, and notified the privacy team per policy.
Related terms
Sources
- Language Models are Few-Shot Learners — Brown et al., 2020paper
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? — Min et al., 2022paper
- In-context Learning and Induction Heads — Olsson et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.