Retrieval-Augmented Generation

A list of numbers (a vector) that represents the meaning of a text, so that similar meanings end up close together.

The high-dimensional space embeddings live in, where the angle or distance between vectors stands for difference in meaning.

A model that reads input text and outputs vector representations of it, rather than generating new text.

Encodes query and documents separately into vectors and compares them: fast and indexable, the backbone of dense retrieval.

Reads the query and one document together and outputs a relevance score: far more precise than vector similarity, too slow to run over a whole corpus.

A score from −1 to 1 measuring the angle between two vectors, ignoring their length: the standard way to compare embeddings.

Multiply two vectors position by position and sum. The cheapest similarity score; identical in ranking to cosine once vectors are unit length.

How many numbers are in each embedding (384, 768, 1024, 1536, 3072). Memory, latency and cost scale linearly with it; quality does not.

Symmetric search matches like with like (ticket to ticket). Asymmetric search matches a short query to a long answer-bearing passage.

A fixed string such as 'query: ' or a task instruction prepended before embedding, telling the model what role the text plays.

Embeddings trained so the first N dimensions work alone: cut 3072 to 256 and keep ~96% of quality. Re-normalize after cutting.

Massive Text Embedding Benchmark: the public leaderboard for embedding models. Good for a shortlist; your own golden set decides.

Training an embedding model further on your own query-passage pairs so domain jargon lands near its meaning. Typically +5–15 points recall in-domain.

Splitting documents into passages small enough to embed and retrieve. Default: ~200–400 tokens on structural boundaries, no overlap.

Repeating the last N tokens of one chunk at the start of the next so ideas cut at a boundary survive. Often unnecessary with structural splits.

Cutting where the topic changes, detected by a drop in embedding similarity between neighbouring sentences, instead of at fixed lengths.

Search over small chunks for precise matching, then hand the LLM the larger parent section that contains them. Retrieve small, return big.

Prepending a short LLM-written note situating each chunk in its document before embedding and indexing. Anthropic measured 35–67% fewer retrieval failures.

Extracting text from PDFs and scans with reading order, headings, tables and figures preserved, so chunking and citations have structure to use.

Optical character recognition: turning images of text (scans, photos, faxes) into machine-readable text. Errors here propagate into every search.

The system that pulls source documents, parses, chunks, embeds and writes them to the index, and keeps it in sync as sources change and delete.

The system of record for ingestion: one row per source document with hash, versions, status, ACL and chunk ids, so the index can be audited and rebuilt.

A fingerprint (e.g. SHA-256) of a document's normalised text, used to skip unchanged content, dedupe copies and make re-processing idempotent.

Change data capture: streaming inserts, updates and deletes from a database's transaction log, so the search index tracks the source within seconds.

A marker recording that a record was deleted, so the deletion propagates through queues, replicas and indexes instead of the record silently reappearing.

Re-processing the whole corpus into a new index (new model, chunker or schema) alongside the live one, then switching traffic once it is verified.

Structured fields stored with each chunk (tenant, ACL, source, date, type, page) used for filtering, permissions, citations and freshness.

Keyword search over an inverted index: match the exact words in the query. Fast, explainable, and the best tool for IDs, codes, names and citations.

The default keyword ranking formula in Lucene, Elasticsearch, OpenSearch and Vespa: rewards rare terms, saturates repeats, normalises length.

Learned sparse retrieval: a BERT-style model outputs weighted vocabulary terms, including expansions, served from an ordinary inverted index.

Search by meaning: embed the query and every chunk as vectors, then return the chunks whose vectors are nearest to the query's.

Run keyword (BM25) and vector search in parallel and merge the lists, usually with RRF, so each covers the other's misses.

Search that matches what the query means rather than the exact words, usually built from embeddings plus a reranker.

Approximate nearest neighbour search: find vectors close to a query without comparing against all of them, trading a little recall for 100–1000× speed.

Hierarchical Navigable Small World: a layered proximity graph searched greedily. The default in-RAM ANN index in pgvector, Qdrant, Lucene, Vespa.

Inverted File index: k-means clusters the vectors into cells; a query scans only the nprobe nearest cells. Cheap to build, easy to shard, pairs with PQ.

Compress a vector by splitting it into m sub-vectors and storing each as a 1-byte codebook index: 3,072 bytes become 64, with fast approximate distances.

Storing embedding values in fewer bits — float16, int8, or 1-bit binary — to cut vector memory 2–32× with a rescoring pass to recover recall.

The share of the true nearest neighbours an ANN index actually returns. Measured against exact search, not human relevance labels.

Restricting search to items whose fields match conditions (tenant, date, ACL, language) — and doing it without wrecking ANN recall.

The cheap, high-recall first stage that narrows millions of items to a few hundred for an expensive reranker to order.

How many results each stage returns. Recall-heavy stages want large k (100+); the context sent to the model wants small k (5–10).

Turning the user's raw message into one or more better search queries: resolving pronouns, expanding jargon, stripping chit-chat.

Splitting a multi-part or multi-hop question into sub-questions, retrieving for each, then combining the evidence.

Hypothetical Document Embeddings: an LLM writes a fake answer, you embed it and search with it — document-to-document matching instead of question-to-document.

Labelling each incoming query by what kind of request it is, so it can be routed to the right pipeline: RAG, SQL, tool call, chit-chat or refusal.

A store that indexes embeddings for ANN search plus metadata filters, upserts, deletes and replication. Often Postgres + pgvector is enough.

Splitting an index across machines so each holds a slice; queries fan out to all shards and merge. Scales size and write throughput, not per-query speed.

Keeping copies of each shard on several nodes for availability and read throughput. Replicas scale QPS; they don't fix an index that's too big.

A slower, more accurate model that re-scores the top 50–200 first-stage candidates so only the best 5–10 reach the LLM.

Reciprocal Rank Fusion: merge ranked lists by summing 1/(k + rank) per document, k = 60. Uses ranks only, so no score calibration is needed.

Combining retriever scores, not ranks: normalise each list then blend, e.g. α·dense + (1−α)·BM25, with α tuned on labelled queries.

Turning ranked chunks into the prompt: dedupe, fit a token budget, order, label with source ids, and add the instructions around them.

2023 models used mid-context facts worse than edge ones. In 2026, single facts are found; multi-hop reasoning and aggregation over long context still degrade.

Making the model answer from supplied sources (retrieved docs, tool results) rather than memory, and checking that it did.

Linking each claim in an answer to the exact source passage that supports it, and measuring whether those citations are correct and complete.

The system declining to answer when evidence is missing or confidence is low, trading coverage for accuracy on what it does answer.

Reusing a stored answer when a new query's embedding is close enough to a past one. Saves cost and latency; risks returning the wrong answer.