Parametric knowledge
Facts an LLM absorbed into its weights during training: broad but frozen at the cutoff, unattributable, and impossible to update or delete one at a time.
Parametric knowledge is what the model remembers from training, stored inside its billions of numeric parameters. It is why a model can name the capital of France without being given any document.
It has limits: it stops at the training date, it cannot tell you where a fact came from, and it knows nothing about your company's private data. The alternative is knowledge you hand it in the Prompt & system prompt, which is what Retrieval-Augmented Generation (RAG) does.
What you remember from school versus the reference book open on the desk: memory is fast and broad, the book is current and citable.
Fine-tuning on company documents to 'teach the model our data' and expecting accurate, citable recall. It learns the style, half-learns the facts, and goes stale at the next policy change.
Factual associations are spread across the weights; feed-forward layers behave partly as key-value memories (Geva et al.). Recall tracks how often a fact appeared in training. Mallen et al. (PopQA) showed closed-book accuracy collapses for long-tail entities, retrieval helps most exactly there, and retrieval can slightly hurt on the most popular facts where the model already knows the answer.
There is no index into the weights, so you cannot update, delete, permission or cite a single fact, and the Knowledge cutoff typically sits 6–12 months before a model's release. Division of labour: weights for language, reasoning patterns and stable world knowledge; retrieval for anything private, fast-changing, citable or subject to access control. Fine-tuning mostly moves behaviour and style; Ovadia et al. (2023) found RAG consistently beat unsupervised fine-tuning for injecting new facts.
Treat the weights as language and reasoning. Retrieve any fact that is private, changes faster than ~6 months, must be cited, or is permissioned. Inject today's date and the product or API version into the system prompt. Require citations on knowledge routes and alert when answers arrive without them.
Financial filings: asked for a mid-cap company's FY2025 revenue, a general model answers with the FY2023 figure, or a plausible round number, with no hedge. With RAG the prompt carries the relevant 10-K table and the model answers '$1.84B, up 9% [10-K FY2025, Item 7]'. For the same question about Apple, closed-book is usually right, which is the popularity effect in action.
This is the 'RAG or fine-tune?' question in disguise. Commit: retrieve facts, tune behaviour. Name the requirements that force retrieval: freshness under the model release cycle, citations, per-user permissions and right-to-delete. The follow-up is 'when would you let the model answer closed-book?': only for stable, high-popularity facts on routes where a wrong answer is cheap, and measured on a slice of the golden set.
Numbers worth knowing
- Cutoff-to-release gap
- 6–12 monthstypical for frontier releases; the model knows nothing after its cutoff
- Closed-book accuracy, long-tail entities
- below 25%least-popular PopQA bins for the models Mallen et al. tested; newer models shift the curve up but keep its shape
- Facts editable per update
- 0 individuallyweights have no per-fact index; changes require retraining or editing techniques that do not scale to thousands of facts
How to decide
| When | Do this |
|---|---|
| Fact is private, changes monthly or faster, or must be cited | Retrieve it; never answer from weights. |
| Fact is stable, popular and a wrong answer is low-cost | Allow closed-book answers on that route, measured on a golden-set slice. |
| You need a style, format or domain vocabulary | Fine-tune or few-shot the behaviour; keep facts in retrieval. |
Cost maths
Fact injection by fine-tuning = runs per year × (compute + eval + review). Assume a LoRA run on an 8B model costs ~$50–200 of rented GPU time plus ~1 engineer-day of evaluation (illustrative, Oct 2026). Weekly content changes mean 52 runs a year and a model that is always up to a week stale. Re-indexing only the changed documents for RAG costs cents of embedding per update and is live in minutes.
Operating it
- More than 2% of answers on a knowledge route carry no citation
- User-reported wrong-fact tickets mention content newer than the model's cutoff
- A new regulated data source (health, finance) is added to the product
- Share of knowledge-route answers with zero citations: alert above 2%
- Mentions of deprecated identifiers (regex list maintained with each release): alert on any
- Golden-set accuracy on the post-cutoff slice: alert at −3 points
When it goes wrong
A developer-docs assistant recommended an SDK method that had been removed in v3, generating a wave of AttributeError support tickets.
Thumbs-down spike on the docs route, and support tickets quoting the assistant's code.
Short questions were routed to a 'fast path' with no retrieval; the model answered from weights trained before v3 shipped.
Every API or version question now goes through docs retrieval, the current SDK version is injected into the system prompt, and a 60-question slice of post-cutoff API changes was added to the golden set.
Related terms
Where it shows up
Sources
- Language Models as Knowledge Bases? — Petroni et al., 2019paper
- Transformer Feed-Forward Layers Are Key-Value Memories — Geva et al., 2021paper
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories — Mallen et al., 2022paper
- Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — Ovadia et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.