Few-shot prompting
Including a handful of worked input-output examples in the prompt so the model copies the format, labels and style you want.
Few-shot prompting means showing the model a few examples of what you want before asking it to do the real thing. 'Zero-shot' is no examples, 'one-shot' is one, 'few-shot' is usually two to ten.
It is one of the simplest ways to get consistent output, especially for exact formats, tricky labels or a particular tone. It works because of In-context learning.
Showing a new barista three finished drinks before they make their first one. A description leaves room for guessing; examples remove it.
Picking only easy, similar examples. The model overfits to their surface details and their majority label, and still fails on the edge cases you needed help with.
Examples shape format, label boundaries and tone more reliably than adjectives. Zhao et al. (2021) documented three biases: majority-label (over-predicting the most frequent label among examples), recency (the last example pulls hardest) and common-token bias. Good practice: cover every label including rare ones, put the hardest distinction in an example, keep format identical, vary surface content, and shuffle order across evaluation runs.
Dynamic few-shot retrieves the k most similar labelled examples per query using embeddings (Liu et al., 2021), which beats a fixed set when inputs vary widely. With modern instruct and Reasoning models & extended thinking, clear instructions plus zero-shot often match few-shot on reasoning, and prescriptive worked solutions can constrain a reasoning model; examples remain most valuable for output format and subtle labelling. For strict machine-readable formats, pair examples with Structured output / decoding.
3–6 examples covering every label (at least one per rare label), one borderline case, consistent format, placed in the cached prefix; dynamic retrieval of k = 4–8 examples from a labelled bank when inputs are heterogeneous; Structured output / decoding for machine-read output.
Legal contract clause classification (standard / non-standard indemnity). A fixed prompt with 5 'standard' and 1 'non-standard' example scored 52% recall on non-standard clauses. Rebalancing to 3+3, with one deliberately borderline clause, lifted recall to 81%; switching to dynamic retrieval of the 6 most similar labelled clauses from a 400-clause bank reached 88%, at ~900 extra input tokens per call.
Show you treat examples as data, not decoration: balanced labels, a borderline case, measured on a Golden set, dynamic retrieval when inputs vary, and fine-tuning when volume makes the tokens expensive. Expect: 'how many examples?' — measure 0, 3, 6, 12 and stop where the curve flattens.
Numbers worth knowing
- Useful example count
- 3–8gains usually flatten after ~8 on classification
- Cost per example
- 100–300 input tokensshort classification/extraction items
- Dynamic retrieval k
- 4–8from a bank of a few hundred labelled items
How to decide
| When | Do this |
|---|---|
| Output format drifts | Add 2–3 format-exact examples plus constrained decoding. |
| Rare labels are missed | Balance examples across labels and include a borderline case. |
| Inputs vary widely by topic | Dynamic few-shot via embedding retrieval. |
Cost maths
Extra cost = calls × examples × tokens/example × input price. 300k contracts/month × 6 × 150 tokens × $3/M = $810/month uncached; dynamic examples cannot share a cached prefix, so caching savings disappear — budget the full price for dynamic few-shot.
Operating it
- Example tokens exceed task tokens on a high-volume route → fine-tune
- Fixed examples underperform on new input types → dynamic retrieval
- Per-label recall spread > 20 points → rebalance examples
- Per-label recall on the golden set — alert if any label < 85%
- Predicted label distribution vs historical base rates — alert on > 2× shift
- Input tokens per call p95 after template edits
When it goes wrong
A contract-review tool marked most non-standard indemnity clauses as standard; reviewers trusted it and missed two unusual liability caps.
A quarterly audit of 200 clauses by in-house counsel measured 52% recall on non-standard clauses.
Five of six few-shot examples were 'standard', producing majority-label bias; nobody measured per-label recall.
Balanced examples, added a borderline case, moved to dynamic retrieval, and added per-label recall to the eval gate with an 85% floor.
Related terms
Sources
- Language Models are Few-Shot Learners — Brown et al., 2020paper
- Calibrate Before Use: Improving Few-Shot Performance of Language Models — Zhao et al., 2021paper
- What Makes Good In-Context Examples for GPT-3? — Liu et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.