Every word, explained.

A transformer trained to predict the next token over trillions of tokens, then tuned to follow instructions: the engine under chat, agents and RAG.

The unit an LLM reads and writes: a word or word fragment. Context limits, latency and price are all metered in tokens, not characters.

The maximum number of tokens, prompt plus output, a model can handle in one call. Anything outside it does not exist for the model.

The full input sent to an LLM: instructions, examples, retrieved context and the user's message. The main lever for controlling behaviour.

Facts an LLM absorbed into its weights during training: broad but frozen at the cutoff, unattributable, and impossible to update or delete one at a time.

When an LLM states something fluent and confident that is false or not supported by its sources.

Continuing to train a pretrained model on your own examples to change its behaviour, format or style. Not a reliable way to add facts.

Retrieval-augmented generation: fetch relevant documents for a question, put them in the prompt, and have the LLM answer from them with citations.

The full collection of documents a retrieval system can search: the source of truth that RAG answers are built from.

The transformer mechanism that lets each token weigh every earlier token in the context when building its representation.

Reusing the model's computed state for an identical prompt prefix across requests: cheaper, faster input for repeated context, minus a cache-write premium.

Constraining generation so output always matches a schema or grammar (e.g. valid JSON) by masking tokens that would break it.

Training a smaller, cheaper student model to imitate a larger teacher model's outputs on your task.

Models or pipelines that handle more than text, such as images, scanned PDFs, audio or video, in the same input or index.

Having an LLM translate a natural-language question into a SQL query, run it under a restricted role, and answer from the result.

RAG where the LLM decides what to retrieve, from which source, and when to search again, in a bounded loop instead of one fixed retrieval step.

Using an LLM with a rubric to grade outputs (faithfulness, relevance, correctness) so evaluation scales beyond human review.

A function built from layers of weighted sums and non-linearities whose weights are learned from examples by gradient descent; the machinery under every LLM.

The learned numbers (weights and biases) inside a model. '70B' means 70 billion of them; they set memory and compute cost, not quality on their own.

The architecture behind nearly all modern LLMs: stacked attention and feed-forward blocks that process every token of a sequence in parallel.

The first and most expensive training phase: a model learns language and world knowledge by predicting the next token across trillions of tokens.

The single task an LLM is trained on and runs at inference: given the text so far, assign a probability to every possible next token.

The text, code and examples a model learns from; mix, quality, recency and licensing decide what it knows and how it behaves.

Empirical power laws: loss falls predictably as parameters, data and compute grow, with diminishing returns per extra 10× of compute.

Supervised fine-tuning on (instruction, good response) pairs that turns a base model that continues text into an assistant that follows requests.

Reinforcement learning from human feedback: train a reward model on human preference comparisons, then optimise the LLM to score highly on it.

Methods like DPO that train directly on chosen-vs-rejected answer pairs, getting RLHF-like alignment without a reward model or RL loop.

Low-Rank Adaptation: fine-tune by training small add-on matrices while the original weights stay frozen; cheap, fast, swappable per task or tenant.

A model adapting to a task purely from instructions and examples in its prompt, with no change to its weights.

Including a handful of worked input-output examples in the prompt so the model copies the format, labels and style you want.

Having a model write intermediate reasoning steps before its final answer, which improves accuracy on multi-step problems at a token cost.

Models trained with RL to think at length before answering, trading latency and output tokens for accuracy on maths, code and multi-step work.

A sampling setting that sharpens (low) or flattens (high) next-token probabilities, trading consistency for variety; it does not change what the model knows.

Nucleus sampling: sample only from the smallest set of tokens whose probabilities add up to p (e.g. 0.9), cutting off the unlikely tail.

The raw scores a model outputs for every token in its vocabulary; softmax turns them into next-token probabilities.

Running a trained model to produce output. For LLMs: a compute-bound prefill over the prompt, then a memory-bound decode loop, one token per step.

Stored attention keys and values for past tokens, so each new token skips recomputing the sequence; at long context it fills GPU memory.

Storing weights (and sometimes activations or KV) in 8 or 4 bits instead of 16 to cut memory and speed up inference, at a quality cost you must measure.

Latency from sending a request until the first output token arrives; the main driver of how responsive a streaming LLM app feels.

Running many requests through the GPU together so one weight read serves many tokens: throughput and cost per token improve, per-stream speed falls.

A cheap draft proposes several tokens and the big model verifies them in one pass: same output distribution, fewer sequential big-model steps.

An architecture where each token is routed to a few of many expert sub-networks, so a huge model computes like a much smaller one per token.

Open-weights models publish their parameters for self-hosting; closed models are API-only. The choice trades control and residency for convenience.

The date after which a model's training data ends; it knows nothing newer unless you supply it in the prompt or through tools.

A list of numbers (a vector) that represents the meaning of a text, so that similar meanings end up close together.

The high-dimensional space embeddings live in, where the angle or distance between vectors stands for difference in meaning.

A model that reads input text and outputs vector representations of it, rather than generating new text.

Encodes query and documents separately into vectors and compares them: fast and indexable, the backbone of dense retrieval.

Reads the query and one document together and outputs a relevance score: far more precise than vector similarity, too slow to run over a whole corpus.

A score from −1 to 1 measuring the angle between two vectors, ignoring their length: the standard way to compare embeddings.

Multiply two vectors position by position and sum. The cheapest similarity score; identical in ranking to cosine once vectors are unit length.

How many numbers are in each embedding (384, 768, 1024, 1536, 3072). Memory, latency and cost scale linearly with it; quality does not.

Symmetric search matches like with like (ticket to ticket). Asymmetric search matches a short query to a long answer-bearing passage.

A fixed string such as 'query: ' or a task instruction prepended before embedding, telling the model what role the text plays.

Embeddings trained so the first N dimensions work alone: cut 3072 to 256 and keep ~96% of quality. Re-normalize after cutting.

Massive Text Embedding Benchmark: the public leaderboard for embedding models. Good for a shortlist; your own golden set decides.

Training an embedding model further on your own query-passage pairs so domain jargon lands near its meaning. Typically +5–15 points recall in-domain.

Splitting documents into passages small enough to embed and retrieve. Default: ~200–400 tokens on structural boundaries, no overlap.

Repeating the last N tokens of one chunk at the start of the next so ideas cut at a boundary survive. Often unnecessary with structural splits.

Cutting where the topic changes, detected by a drop in embedding similarity between neighbouring sentences, instead of at fixed lengths.

Search over small chunks for precise matching, then hand the LLM the larger parent section that contains them. Retrieve small, return big.

Prepending a short LLM-written note situating each chunk in its document before embedding and indexing. Anthropic measured 35–67% fewer retrieval failures.

Extracting text from PDFs and scans with reading order, headings, tables and figures preserved, so chunking and citations have structure to use.

Optical character recognition: turning images of text (scans, photos, faxes) into machine-readable text. Errors here propagate into every search.

The system that pulls source documents, parses, chunks, embeds and writes them to the index, and keeps it in sync as sources change and delete.

The system of record for ingestion: one row per source document with hash, versions, status, ACL and chunk ids, so the index can be audited and rebuilt.

A fingerprint (e.g. SHA-256) of a document's normalised text, used to skip unchanged content, dedupe copies and make re-processing idempotent.

Change data capture: streaming inserts, updates and deletes from a database's transaction log, so the search index tracks the source within seconds.

A marker recording that a record was deleted, so the deletion propagates through queues, replicas and indexes instead of the record silently reappearing.

Re-processing the whole corpus into a new index (new model, chunker or schema) alongside the live one, then switching traffic once it is verified.

Structured fields stored with each chunk (tenant, ACL, source, date, type, page) used for filtering, permissions, citations and freshness.

Keyword search over an inverted index: match the exact words in the query. Fast, explainable, and the best tool for IDs, codes, names and citations.

The default keyword ranking formula in Lucene, Elasticsearch, OpenSearch and Vespa: rewards rare terms, saturates repeats, normalises length.

Learned sparse retrieval: a BERT-style model outputs weighted vocabulary terms, including expansions, served from an ordinary inverted index.

Search by meaning: embed the query and every chunk as vectors, then return the chunks whose vectors are nearest to the query's.

Run keyword (BM25) and vector search in parallel and merge the lists, usually with RRF, so each covers the other's misses.

Search that matches what the query means rather than the exact words, usually built from embeddings plus a reranker.

Approximate nearest neighbour search: find vectors close to a query without comparing against all of them, trading a little recall for 100–1000× speed.

Hierarchical Navigable Small World: a layered proximity graph searched greedily. The default in-RAM ANN index in pgvector, Qdrant, Lucene, Vespa.

Inverted File index: k-means clusters the vectors into cells; a query scans only the nprobe nearest cells. Cheap to build, easy to shard, pairs with PQ.

Compress a vector by splitting it into m sub-vectors and storing each as a 1-byte codebook index: 3,072 bytes become 64, with fast approximate distances.

Storing embedding values in fewer bits — float16, int8, or 1-bit binary — to cut vector memory 2–32× with a rescoring pass to recover recall.

The share of the true nearest neighbours an ANN index actually returns. Measured against exact search, not human relevance labels.

Restricting search to items whose fields match conditions (tenant, date, ACL, language) — and doing it without wrecking ANN recall.

The cheap, high-recall first stage that narrows millions of items to a few hundred for an expensive reranker to order.

How many results each stage returns. Recall-heavy stages want large k (100+); the context sent to the model wants small k (5–10).

Turning the user's raw message into one or more better search queries: resolving pronouns, expanding jargon, stripping chit-chat.

Splitting a multi-part or multi-hop question into sub-questions, retrieving for each, then combining the evidence.

Hypothetical Document Embeddings: an LLM writes a fake answer, you embed it and search with it — document-to-document matching instead of question-to-document.

Labelling each incoming query by what kind of request it is, so it can be routed to the right pipeline: RAG, SQL, tool call, chit-chat or refusal.

A store that indexes embeddings for ANN search plus metadata filters, upserts, deletes and replication. Often Postgres + pgvector is enough.

Splitting an index across machines so each holds a slice; queries fan out to all shards and merge. Scales size and write throughput, not per-query speed.

Keeping copies of each shard on several nodes for availability and read throughput. Replicas scale QPS; they don't fix an index that's too big.

A slower, more accurate model that re-scores the top 50–200 first-stage candidates so only the best 5–10 reach the LLM.

Reciprocal Rank Fusion: merge ranked lists by summing 1/(k + rank) per document, k = 60. Uses ranks only, so no score calibration is needed.

Combining retriever scores, not ranks: normalise each list then blend, e.g. α·dense + (1−α)·BM25, with α tuned on labelled queries.

Turning ranked chunks into the prompt: dedupe, fit a token budget, order, label with source ids, and add the instructions around them.

2023 models used mid-context facts worse than edge ones. In 2026, single facts are found; multi-hop reasoning and aggregation over long context still degrade.

Making the model answer from supplied sources (retrieved docs, tool results) rather than memory, and checking that it did.

Linking each claim in an answer to the exact source passage that supports it, and measuring whether those citations are correct and complete.

The system declining to answer when evidence is missing or confidence is low, trading coverage for accuracy on what it does answer.

Reusing a stored answer when a new query's embedding is close enough to a past one. Saves cost and latency; risks returning the wrong answer.

A system where the LLM directs its own steps and tool calls in a loop until the goal is met or a stop condition fires (Anthropic, Dec 2024).

Workflows run LLMs through code paths you wrote; agents let the model pick the path. Anthropic's 2024 split; trade predictability for flexibility.

The model emits a structured call (tool name + JSON arguments); your application validates and runs it, then returns the result to the model.

The core cycle of every agent: model decides, host runs a tool, result is appended, repeat until a final answer or a stop condition.

Having the agent write out the steps for a goal before acting, then executing and re-planning when a tool result contradicts the plan.

The model critiques its output or trajectory and revises it; it helps when the critique uses an external signal (tests, validators), not self-review alone.

A fixed sequence of LLM calls, each feeding the next, with programmatic gates between steps — the first workflow pattern in Anthropic's 2024 list.

Several LLM agents with separate contexts, instructions and tools cooperating on one task: strong for parallel breadth, ~15× chat tokens.

A lead model splits a task into sub-tasks at run time, dispatches them to workers (calls or agents), then merges results — Anthropic's 4th pattern.

One agent transfers control of a conversation or task to another agent or a human, passing a structured briefing and swapping tools and permissions.

Model Context Protocol: an open JSON-RPC standard connecting AI apps (hosts) to servers exposing tools, resources and prompts. Linux Foundation since Dec 2025.

How an agent keeps information beyond one context window: working memory in context, plus long-term stores it writes to and retrieves from.

Deciding exactly what enters the model's context at each step — instructions, tools, retrieved data, history — to maximise signal per token.

An agent operating a graphical interface from screenshots via mouse and keyboard actions — the fallback for software with no API.

An isolated environment where an agent's code and actions run with scoped files, network and credentials, so mistakes can't reach real systems.

Checks around an LLM or agent that block, modify or escalate unsafe inputs, outputs and actions — layered, with the critical ones in code.

A person reviews, approves or corrects an AI's work at chosen points — usually before irreversible or high-stakes actions.

Rules that end an agent loop: task done, or a cap on turns, tokens, time, spend, repeated actions or errors — each with a logged reason.

Running agent workflows so progress is checkpointed and they resume after crashes, deploys or long waits without redoing completed side effects.

Evaluating an agent's whole path — tool choices, arguments, order, safety, efficiency — and its end state, over repeated trials (pass^k).

Designing an agent's tools — names, descriptions, parameters, outputs, errors — so the model picks and uses them correctly and cheaply.

What an agent run costs in tokens, money and time. It grows quadratically with turns unless cached, so manage it as a distribution per success.

A fixed, versioned set of realistic inputs with known-good answers or relevant documents, used to score every change the same way.

The code that runs a system against a golden set, scores outputs with metrics and judges, and reports diffs, ideally on every pull request.

Of all the relevant documents for a query, the fraction that appear in the top k results. The ceiling on what a RAG system can answer.

Of the results you returned, the fraction that are relevant. Precision@k measures how much of the top k is useful versus noise.

Normalised Discounted Cumulative Gain: graded-relevance ranking score that rewards the best results near the top. 1.0 = ideal order.

Mean Reciprocal Rank: average of 1/rank of the first relevant result per query. Answers 'how far down is the first good hit?'

Share of an answer's claims that the provided context supports. Measures grounding, not truth: a faithful answer from a wrong doc is still wrong.

RAG metric: are the relevant chunks ranked at the top of the context? Rank-weighted precision, so noise above the evidence costs more.

Latency at a percentile: p50 is the typical request, p95/p99 the slow tail. SLOs use the tail because averages hide the users who wait.

Queries per second: request rate a system handles. For LLMs, convert QPS to concurrency and tokens/s; that is what sizes GPUs and bills.

Service Level Objective: a target for a measured indicator over a window, e.g. 99.5% of answers under 5 s over 30 days. Its gap is the error budget.

Replace one pipeline stage with perfect input (e.g. gold passages) and measure the ceiling. The gap shows which stage to fix first.

Reading real failures, labelling why each happened, and counting the categories, so you fix the biggest cause instead of guessing.

Production behaviour shifting from what you evaluated: user inputs, corpus, provider model, or what 'correct' means changes under you.

Decide which number a change should move, build the cases that measure it, then change the prompt, model or retriever and read the delta with error bars.

Offline eval scores a fixed test set before release; online eval measures real users after it. Offline is the gate, online is the verdict.

A written scoring guide: one observable criterion per line, a small scale (usually pass/fail), and an anchored example for each score.

Grading by asking which of two answers is better, not scoring each alone; more reliable for subjective quality, but needs order swaps to cancel position bias.

People labelling model outputs against a rubric; the ground truth that LLM judges are calibrated against, and the most expensive data in the eval stack.

How often independent graders agree beyond chance, usually Cohen's kappa; below ~0.6 your labels, and any judge validated against them, are unreliable.

A public, fixed test set and scoring rule (MMLU, GPQA, SWE-bench) for comparing models; a coarse filter for model choice, never a substitute for your own eval.

When test items (or close paraphrases) leaked into a model's training data, so its score measures memory instead of skill.

A CI check that runs the eval suite on every prompt, model or config change and blocks the merge or rollout if quality drops beyond the measured noise.

Randomly split live users between the current and new version, then compare a pre-chosen outcome metric with a significance test to decide what ships.

Whether an eval or A/B delta is bigger than the noise; report confidence intervals, use paired tests on shared items, and size n before you run.

Model-generated inputs, labels or conversations that fill eval or training gaps; fast to make, trustworthy only after filtering, dedupe and human spot checks.

The loop that turns production traces and feedback into labelled failures, new eval cases and fixes, so each week of traffic makes the system measurably better.

Deliberately attacking your own AI system (jailbreaks, prompt injection, data extraction, unsafe tool use) to find failures before attackers or users do.

An input crafted to make a model ignore its safety training or system instructions: role-play framing, encoded text, many-shot examples, adversarial suffixes.

Detecting and masking personal data (names, emails, IDs, health details) before it reaches a model, a log or a vendor; recall matters more than precision.

A proxy between your apps and model providers that centralises auth, routing, fallback, rate limits, budgets, caching, redaction and logging.

Capping requests and tokens per key, tenant or user per time window, usually with token buckets, so one caller cannot exhaust shared provider quota or budget.

Routing a request to a backup model or provider when the primary errors, times out or is overloaded; only safe if the backup has passed the same evals.

Treating prompts as immutable, versioned bundles pinned to a model snapshot and logged on every trace, so any output traces back to exactly what produced it.

Total spend to answer one request: input, cache writes and reads, output and reasoning tokens, retrieval, reranking, tool calls and retries.

A provider retiring a model snapshot on a set date; every dependent prompt, eval and fallback must be migrated and re-validated before requests start failing.

Whether stated confidence matches actual accuracy: of answers given 80% confidence, about 80% should be right. Needed for abstention and routing thresholds.

Running an operation twice leaves the same state as running it once. It is the property that makes retries and at-least-once queues safe.

Pushing back on producers when a consumer is saturated: bounded queues, 429s and load shedding instead of unbounded buffering until latency or memory collapses.

A side queue where messages land after N failed attempts, so one poison message stops blocking the pipeline and can be inspected and replayed later.

Wait longer after each failed retry (1 s, 2 s, 4 s…) plus random jitter, so clients don't hammer a struggling service in sync.

Run the new version (green) beside the old (blue), verify it on real infra, then switch all traffic at once, keeping blue warm for instant rollback.

Writing the same change to two stores, e.g. Postgres and a vector index. Done naively it drifts; the outbox pattern plus CDC makes it reliable.

Copy real production requests to a new version whose responses users never see, then compare outputs, latency and cost before risking live traffic.

Release to a small slice of real traffic first (1%, then 5%, 25%…), compare against the old version, and roll back automatically if metrics degrade.

A runtime switch that turns code paths, models or prompts on or off per user, tenant or percentage without a redeploy.

A stable name (e.g. products-read) that points to a physical index, so you can rebuild a new index and repoint readers atomically with no client change.

Background rewriting of an index's segments to drop deleted and overwritten entries, reclaiming space and restoring search speed and recall.

Being able to answer new questions about production from its telemetry. For LLM systems: traces per request with prompts, tool calls, tokens, cost and quality.

Time to live: how long a cached or stored item is valid before it expires. Too long serves stale data; too short throws away savings.

A scheduled job that compares the source of truth with derived stores (index, cache, ACLs) and repairs drift that events and retries missed.

When a dependency fails, serve a reduced but useful response (smaller model, cached answer, sources only) instead of an error page.

A blameless written review after an incident: timeline, impact, root causes, what went well, and owned action items with due dates.

Serving many customers from shared infrastructure while keeping their data, quotas and performance isolated. Isolation tier is a cost vs risk choice.

One tenant's heavy load (a bulk import, a runaway agent) degrading latency or quota for everyone sharing the same infrastructure.

Access control list: who may see a document. In RAG, ACLs must be enforced at retrieval time or the model will quote what the user can't open.

When a security check can't run (auth service down, classifier timeout), deny by default. The opposite, fail-open, keeps serving and skips the check.

Text that the model reads as instructions: typed by a user (direct) or hidden in a page, email or doc the agent reads (indirect). No filter fully fixes it.

Getting data out of a system without authorisation. For agents: via a tool call, a fetched URL, or a rendered markdown image carrying data in its query string.

Give each user, service and agent only the permissions its current task needs, for only as long as it needs them. Limits damage when it's tricked or buggy.

How much breaks when one thing fails or is compromised: how many users, tenants, records or actions. Design to keep it small and known.

Stacking independent controls so one failing (and each will sometimes) doesn't mean compromise. For LLM agents, no single layer is reliable alone.

Situation, Task, Action, Result: a structure for behavioural interview answers. Spend most of the time on what you did and what changed, with numbers.

Choosing one option knowing what it costs you on another axis. Senior answers name the axes, the deciding constraint, and when they'd switch.

A senior IC level whose scope spans teams: sets technical direction, de-risks big bets, and multiplies others, mostly through writing and influence.