One line per term, the default to reach for, and the number to remember. Print it or save it as a PDF.
A list of numbers (a vector) that represents the meaning of a text, so that similar meanings end up close together.
Default: bge-m3 (568M params, 1,024 dims, 8,192-token input) self-hosted, or a hosted embedding model truncated to 1,024 dims; L2-normalise at write; store model@version with every vector; hybrid with BM25; re-embed via a new index behind an Index alias.
Dimensions: 384–4,096 (common production models; 768–1,024 is the default sweet spot)
The high-dimensional space embeddings live in, where the angle or distance between vectors stands for difference in meaning.
Default: One model per index, version recorded; L2-normalise and use inner product; exact search (FAISS flat or pgvector without an index) below ~1M vectors, HNSW above; thresholds calibrated per model on 300–1,000 labelled pairs; UMAP plots for exploration only.
Random-vector cosine spread: std ≈ 1/√d (≈0.03 at d = 1,024; real models are anisotropic and sit higher)
A model that reads input text and outputs vector representations of it, rather than generating new text.
Default: bge-m3 for multilingual or long chunks; all-MiniLM-L6-v2 when CPU-only and chunks are short; served with Text Embeddings Inference or Sentence-Transformers, batch 32–128, FP16 on GPU; an ingest assertion that token length ≤ max_seq_length.
Encoder sizes: 22M–7B parameters (production embedding models, Oct 2026)
Encodes query and documents separately into vectors and compares them: fast and indexable, the backbone of dense retrieval.
Default: bge-m3 or a hosted embedding model; documents embedded at ingest, queries at request time with the model's documented prefix; HNSW (M = 16, ef_construction = 200, ef_search tuned to recall ≥ 0.95 vs exact); top 50–100 candidates handed to a reranker.
Query encode: 5–15 ms (100M–600M encoder on a GPU)
Reads the query and one document together and outputs a relevance score: far more precise than vector similarity, too slow to run over a whole corpus.
Default: bge-reranker-v2-m3 (568M, multilingual) self-hosted on one GPU, or a hosted rerank API; rerank the top 50 from hybrid search, keep 5–8; max length 512 tokens per pair; batch all candidates in one call; MiniLM-L6 cross-encoder when CPU-only or under 100 ms total budget.
Rerank latency, small model: 100–300 ms per 100 candidates (MiniLM-L6 (22M) on one A10G, 300-token chunks)
A score from −1 to 1 measuring the angle between two vectors, ignoring their length: the standard way to compare embeddings.
Default: L2-normalise every vector at write time and at query time; index with inner product (pgvector vector_ip_ops, FAISS IndexFlatIP/HNSW-IP); follow the model card if it specifies dot product on raw vectors; calibrate thresholds per model on 500–1,000 labelled pairs.
Unit-vector identity: ‖a − b‖² = 2 − 2cos(a, b) (holds only when both vectors are L2-normalised)
Multiply two vectors position by position and sum. The cheapest similarity score; identical in ranking to cosine once vectors are unit length.
Default: L2-normalize every vector at write and query time, assert the norm in the ingestion path, and index with inner product: pgvector vector_ip_ops (the <#> operator returns the negative inner product, so order ascending), FAISS METRIC_INNER_PRODUCT, Qdrant Dot (or Cosine, which normalizes on insert).
Bytes read per 1024-dim candidate: 4 KB float32, 1 KB int8, 128 B binary (uncompressed vector, no index overhead)
How many numbers are in each embedding (384, 768, 1024, 1536, 3072). Memory, latency and cost scale linearly with it; quality does not.
Default: Start at 1024 dims: bge-m3 self-hosted, or text-embedding-3-large / gemini-embedding-001 truncated to 1024, or voyage-3.5 at its 1024 default.
Raw float32 storage: 4 KB per 1024-dim vector; 4.1 GB per 1M (before index overhead)
Symmetric search matches like with like (ticket to ticket). Asymmetric search matches a short query to a long answer-bearing passage.
Default: For RAG, use an asymmetric model and always pass the role: Cohere input_type, Voyage input_type, E5/Nomic prefixes, or the query instruction for bge and Qwen3-Embedding.
Typical MS MARCO query vs passage length: ~6 words vs ~55 words (MS MARCO passage ranking)
A fixed string such as 'query: ' or a task instruction prepended before embedding, telling the model what role the text plays.
Default: One shared embedding client that owns model id, revision, query prefix and document prefix as a single versioned config; every chunk stores the config hash; ingestion refuses to write if hashes disagree.
Prefix cost: 2–20 tokens per input (fixed role strings vs free-text task instruction)
Embeddings trained so the first N dimensions work alone: cut 3072 to 256 and keep ~96% of quality. Re-normalize after cutting.
Default: text-embedding-3-large or gemini-embedding-001 at 1024 dims for the main index; 256-dim prefix plus rescoring when RAM dominates.
Quality at 256 of 3072 dims: 62.0 vs 64.6 MTEB (≈96%) (text-embedding-3-large, OpenAI-reported)
Massive Text Embedding Benchmark: the public leaderboard for embedding models. Good for a shortlist; your own golden set decides.
Default: Shortlist 3–5 models from the MTEB retrieval tab filtered by language and size, always including one API model (voyage-3.5, text-embedding-3-large, gemini-embedding-001 or Cohere embed-v4) and one self-hostable model (bge-m3 or Qwen3-Embedding-0.6B).
Original MTEB scope: 58 datasets, 8 task types, 112 languages (Muennighoff et al., 2022)
Training an embedding model further on your own query-passage pairs so domain jargon lands near its meaning. Typically +5–15 points recall in-domain.
Default: Sentence Transformers v3+ trainer, base model bge-base-en-v1.5 or bge-m3 (or Qwen3-Embedding-0.6B for multilingual), CachedMultipleNegativesRankingLoss, batch 512–1,024, 2 hard negatives per pair from ranks 10–50 filtered by a cross-encoder, lr 2e-5, 1–2 epochs, golden set split by document.
Typical in-domain gain: +5–15 points recall@10 (jargon-heavy domain, 10k–50k pairs, vs same base model)
Splitting documents into passages small enough to embed and retrieve. Default: ~200–400 tokens on structural boundaries, no overlap.
Default: Parse to Markdown first (Docling or Unstructured), then a recursive splitter (LangChain RecursiveCharacterTextSplitter.from_tiktoken_encoder or LlamaIndex SentenceSplitter) at 256–400 tokens of the embedding model's tokenizer, 0 overlap, headings prepended as a breadcrumb, tables and code blocks kept whole up to 800–1,000 tokens.
Recall vs size: 88.1% @200 · 89.5% @400 · 85.4% @800(+400 overlap) (Chroma 2024, text-embedding-3-large, 5 chunks)
Repeating the last N tokens of one chunk at the start of the next so ideas cut at a boundary survive. Often unnecessary with structural splits.
Default: 0 overlap with structure-aware splitting plus read-time expansion (neighbouring chunks or parent section).
Storage multiplier: 1 / (1 − overlap fraction): 1.25× at 20%, 2× at 50% (fixed-length windows)
Cutting where the topic changes, detected by a drop in embedding similarity between neighbouring sentences, instead of at fixed lengths.
Default: LangChain SemanticChunker or LlamaIndex SemanticSplitterNodeParser with a small fast embedder (bge-small-en-v1.5 or all-MiniLM-L6-v2) for breakpoint detection, buffer of 1 sentence each side, 90–95th percentile threshold, 100-token minimum and 500-token maximum; then embed the final chunks with your main model.
Extra ingest embedding calls: 5–20× chunk count (one call per sentence (window))
Search over small chunks for precise matching, then hand the LLM the larger parent section that contains them. Retrieve small, return big.
Default: Children of 150–250 tokens indexed with parent_id; parents = heading sections capped at 2,000 tokens (split larger ones); retrieve 30 children, group, return top 3 parents ordered best-first.
Context per answer: ~1.5k (children) → 4–6k tokens (3 parents) (200-token children, 1.5–2k parents)
Prepending a short LLM-written note situating each chunk in its document before embedding and indexing. Anthropic measured 35–67% fewer retrieval failures.
Default: Step 1, always: prepend title › section path › date to every chunk.
Retrieval failure reduction: 35% (embed) · 49% (+BM25) · 67% (+rerank) (Anthropic benchmark, top-20, Sept 2024)
Extracting text from PDFs and scans with reading order, headings, tables and figures preserved, so chunking and citations have structure to use.
Default: Docling (open source) to Markdown with page numbers and bounding boxes; tables exported as Markdown or HTML and chunked whole; page headers and footers dropped.
Docling throughput: ~1–2 s/page CPU, sub-second on GPU (text-layer PDFs with table structure on (estimate; measure))
Optical character recognition: turning images of text (scans, photos, faxes) into machine-readable text. Errors here propagate into every search.
Default: Preprocess (deskew, 300 DPI, binarise), Tesseract 5 or PaddleOCR with word confidences, escalate pages with mean confidence <80 to a managed OCR service, keep bounding boxes and engine version per page, and send identifier fields (policy numbers, MRNs, invoice ids) with confidence <90 to human review.
Tesseract CER: ~1–2% clean 300-DPI print; 5–10%+ on faxes/photos (printed Latin script, no handwriting)
The system that pulls source documents, parses, chunks, embeds and writes them to the index, and keeps it in sync as sources change and delete.
Default: Event-driven per-document jobs on SQS or Kafka with stateless workers; Docling for parsing; SHA-256 content hash on normalised text; deterministic chunk ids; embedding batches of 64–256; upsert then delete stale chunk ids; registry write last; DLQ after 3 retries with exponential backoff; nightly reconciliation job.
Unchanged-content events skipped by hash: 50–80% (wikis and CMSs where metadata edits dominate)
The system of record for ingestion: one row per source document with hash, versions, status, ACL and chunk ids, so the index can be audited and rebuilt.
Default: A Postgres table keyed by (tenant_id, doc_id) with the columns above, indexes on status and embed_model, row written after the index upsert (or via a transactional outbox), soft deletes with deleted_at, and raw source snapshots in object storage keyed by content hash so the index can be rebuilt without re-crawling.
Row size: ~1–2 KB (with ~20 chunk ids and ACL list)
A fingerprint (e.g. SHA-256) of a document's normalised text, used to skip unchanged content, dedupe copies and make re-processing idempotent.
Default: SHA-256 over normalised extracted text; document hash in the registry; chunk embedding cache keyed by sha256(chunk_text) + embed_model + prefix_version + chunker_version; MinHash LSH (datasketch, 128 permutations, Jaccard 0.8–0.85) for near-duplicate collapse on web or marketplace corpora.
Truly changed share on nightly re-crawl: 1–5% (catalogs and docs sites after normalisation)
Change data capture: streaming inserts, updates and deletes from a database's transaction log, so the search index tracks the source within seconds.
Default: Debezium (Postgres pgoutput) to Kafka or a managed equivalent, topic partitioned by primary key, max_slot_wal_keep_size set, Kafka retention ≥ 7 days, consumer applying events only when source LSN/version is newer than the stored one, metadata-only updates skipping re-embed, nightly reconciliation against a table snapshot.
Source-to-searchable latency: seconds (p95 5–30 s) (CDC + streaming consumer, excluding batch embedding waits)
A marker recording that a record was deleted, so the deletion propagates through queues, replicas and indexes instead of the record silently reappearing.
Default: Deletes as events (Kafka tombstones or explicit op: d), registry deleted_at + chunk ids, delete by id (never by fuzzy filter), purge semantic-cache entries by cited doc id, nightly orphan reconciliation, compaction/vacuum on a deleted-ratio threshold, and a restore runbook that replays registry deletions before serving.
Kafka tombstone retention: delete.retention.ms = 24 h default (compacted topics; raise if consumers can lag longer)
Re-processing the whole corpus into a new index (new model, chunker or schema) alongside the live one, then switching traffic once it is verified.
Default: Versioned indexes behind an alias (docs → docs_v7); backfill from the registry's cached text, sharded by doc-id hash with per-shard checkpoints; provider Batch API when latency allows; record CDC/stream position at start and replay; gate the switch on count parity (±0.1%), golden-set recall no worse than −0.5 points, and 2–3 days of shadow traffic; keep v1 for 7–14 days.
Duration: tokens ÷ throughput, ×1.5–2 for retries (e.g. 10.6B tokens at 3M tokens/min ≈ 2.5 days)
Structured fields stored with each chunk (tenant, ACL, source, date, type, page) used for filtering, permissions, citations and freshness.
Default: Every chunk carries tenant_id, acl, doc_id, chunk_id, source, doc_type, section_path, page, updated_at and embed_model; payload indexes on tenant_id, acl and any field used in filters; tenant and ACL filters injected server-side from the authenticated principal; volatile fields (price, stock, ACL) in metadata only.
Metadata per chunk: ~200 B–1 KB (15–20 fields, short ACL lists)
Keyword search over an inverted index: match the exact words in the query. Fast, explainable, and the best tool for IDs, codes, names and citations.
Default: Elasticsearch or OpenSearch with BM25 (k1=1.2, b=0.75 defaults), a standard analyzer on prose fields and a separate code-aware or keyword sub-field for identifiers.
Query latency, 3-term query: 5–20 ms p50 (10M passages, one 8-vCPU Elasticsearch node, warm page cache)
The default keyword ranking formula in Lucene, Elasticsearch, OpenSearch and Vespa: rewards rare terms, saturates repeats, normalises length.
Default: Engine defaults k1=1.2, b=0.75; per-field similarity only after an offline eval shows a gain.
Lucene defaults: k1 = 1.2, b = 0.75 (Elasticsearch, OpenSearch, Solr)
Learned sparse retrieval: a BERT-style model outputs weighted vocabulary terms, including expansions, served from an ordinary inverted index.
Default: On Elasticsearch: ELSER into a sparse_vector field, fused with BM25 and a dense leg via the rrf retriever.
BEIR zero-shot nDCG@10: ≈0.47–0.50 average (SPLADE v2 / SPLADE++ checkpoints vs ≈0.43 for BM25)
Search by meaning: embed the query and every chunk as vectors, then return the chunks whose vectors are nearest to the query's.
Default: Open-weights bge-m3 (568M, 1024-d, multilingual) or Qwen3-Embedding-0.6B self-hosted on one L4; or an API model (OpenAI text-embedding-3-small, Cohere Embed, Voyage) when you cannot host.
Query embed latency: 5–20 ms self-hosted, 50–200 ms API (single query, ≤64 tokens, 0.5B-class model on L4)
Run keyword (BM25) and vector search in parallel and merge the lists, usually with RRF, so each covers the other's misses.
Default: BM25 top 100 + dense top 100, RRF with k=60, cross-encoder (bge-reranker-v2-m3, 568M) on the fused top 50, 5–10 chunks to the LLM.
RRF constant: k = 60 (Cormack et al. 2009; Elasticsearch default rank_constant)
Search that matches what the query means rather than the exact words, usually built from embeddings plus a reranker.
Default: Hybrid retrieval (BM25 + bge-m3 or text-embedding-3-small) to 200 candidates, structured filters from a lightweight query parser, bge-reranker-v2-m3 on the top 50, sentence-level highlighting from the reranker's best sentence.
BERT rerank gain over BM25: MRR@10 0.187 → 0.365 (MS MARCO passage dev (Nogueira & Cho 2019))
Approximate nearest neighbour search: find vectors close to a query without comparing against all of them, trading a little recall for 100–1000× speed.
Default: ≤ 100k vectors: exact search.
HNSW query latency: 1–3 ms (10M × 768-d in RAM, ef_search 100, recall@10 ≈ 0.95–0.98)
Hierarchical Navigable Small World: a layered proximity graph searched greedily. The default in-RAM ANN index in pgvector, Qdrant, Lucene, Vespa.
Default: M=16, ef_construction=64–128, ef_search=100 (≥ 5× k), cosine via normalised vectors + inner product.
Link overhead: ~128–200 B per vector at M=16 (4-byte neighbour ids, 2·M links on layer 0)
Inverted File index: k-means clusters the vectors into cells; a query scans only the nprobe nearest cells. Cheap to build, easy to shard, pairs with PQ.
Default: FAISS: nlist ≈ 4√N–16√N (IVF4096,Flat at 1M, IVF65536,PQ64 at 100M+), train on ≥ 30 × nlist random vectors, nprobe swept to hit recall@10 ≥ 0.95 after rescoring.
Distance computations per query: nlist + nprobe·N/nlist (flat lists; minimised near nlist ≈ √N)
Compress a vector by splitting it into m sub-vectors and storing each as a 1-byte codebook index: 3,072 bytes become 64, with fast approximate distances.
Default: FAISS OPQ{m},IVF{nlist},PQ{m} with m = D/8–D/12 (64–96 at 768-d), 8-bit codes, train on ≥ 256 × 256 vectors per subspace sample (~1M vectors is safe), rescore the top 200–1,000 with float16 vectors.
Compression: 48× (3,072 B → 64 B) (768-d float32, PQ64 with 8-bit codes)
Storing embedding values in fewer bits — float16, int8, or 1-bit binary — to cut vector memory 2–32× with a rescoring pass to recover recall.
Default: float16 storage by default (pgvector halfvec).
Memory reduction: 2× / 4× / 32× (float16 / int8 / binary vs float32)
The share of the true nearest neighbours an ANN index actually returns. Measured against exact search, not human relevance labels.
Default: Nightly job: 1,000–2,000 sampled production queries, exact top-k via brute force, recall@10 reported overall and per segment (tenant size bucket, filter vs no filter, data age).
Typical production target: recall@10 ≥ 0.95 (with a reranker and fusion downstream)
Restricting search to items whose fields match conditions (tenant, date, ACL, language) — and doing it without wrecking ANN recall.
Default: Payload/metadata index on every filtered field (Qdrant create_payload_index, B-tree in Postgres).
Post-filter expected matches: ef_search × selectivity (e.g. 100 × 1% = 1 row for k=10)
The cheap, high-recall first stage that narrows millions of items to a few hundred for an expensive reranker to order.
Default: Hybrid (BM25 + dense) to 100–200 fused candidates; rerank 50–100 with a cross-encoder; pass 5–10 to the LLM.
Stage-1 recall target: ≥ 0.9 at the handoff N (labelled golden set; below this the reranker is starved)
How many results each stage returns. Recall-heavy stages want large k (100+); the context sent to the model wants small k (5–10).
Default: Retrieval k = 100 (each hybrid leg), rerank top 50, reranker score threshold tuned on the golden set (start at 0.2–0.3), cap 8 chunks / ~6k tokens in context, near-duplicate removal before the cap.
Mid-context accuracy drop: 20+ points (answer in the middle of 20 docs vs first position (Liu et al. 2023))
Turning the user's raw message into one or more better search queries: resolving pronouns, expanding jargon, stripping chit-chat.
Default: Standalone-question rewrite for every turn after the first, using a small model with JSON output and a 2-second timeout falling back to the raw query.
Follow-ups needing contextualisation: 20–40% of turns (multi-turn assistant traffic (typical sampled logs))
Splitting a multi-part or multi-hop question into sub-questions, retrieving for each, then combining the evidence.
Default: Gate on intent (comparison, multi-hop).
Added latency, parallel: 300–800 ms (one small-model planner call + concurrent retrieval)
Hypothetical Document Embeddings: an LLM writes a fake answer, you embed it and search with it — document-to-document matching instead of question-to-document.
Default: Off by default.
Added latency: 300–1,500 ms (one ~120–150-token generation on a small hosted model)
Labelling each incoming query by what kind of request it is, so it can be routed to the right pipeline: RAG, SQL, tool call, chit-chat or refusal.
Default: Embedding + logistic-regression classifier (200–500 examples/intent) with a confidence threshold of 0.7; LLM (small model, JSON schema) for the low-confidence tail; unknown falls back to hybrid RAG; all routing decisions logged with confidence.
Embedding classifier latency: 2–10 ms (small encoder on CPU + linear head)
A store that indexes embeddings for ANN search plus metadata filters, upserts, deletes and replication. Often Postgres + pgvector is enough.
Default: pgvector (0.8+) in the existing Postgres up to ~10M vectors: halfvec, HNSW m=16 / ef_construction=64–128, iterative scans for filters.
pgvector comfortable range: ≲10M vectors per instance (768–1024-d halfvec HNSW, index fits in RAM alongside OLTP working set)
Splitting an index across machines so each holds a slice; queries fan out to all shards and merge. Scales size and write throughput, not per-query speed.
Default: Elasticsearch/OpenSearch: 10–50 GB per shard, primaries sized for 12–18 months of growth, time- or alias-based indices for rollover.
Shard size: 10–50 GB, < 200M docs (Elasticsearch sizing guidance)
Keeping copies of each shard on several nodes for availability and read throughput. Replicas scale QPS; they don't fix an index that's too big.
Default: 1 replica minimum in production (2 copies), across availability zones; 2 replicas for indices that must survive a node loss during maintenance.
Read scaling: ~linear in copies (until coordinator/network saturates)
A slower, more accurate model that re-scores the top 50–200 first-stage candidates so only the best 5–10 reach the LLM.
Default: Hybrid retrieval → top 100 → bge-reranker-v2-m3 (FP16, max length 512, batch 32) on a GPU behind Hugging Face TEI or Infinity → keep top 5–8.
MiniLM-L6 cross-encoder latency: 300–600 ms CPU, 30–60 ms GPU (100 pairs × 256 tokens; 8 vCPU ONNX vs one L4/A10G)
Reciprocal Rank Fusion: merge ranked lists by summing 1/(k + rank) per document, k = 60. Uses ranks only, so no score calibration is needed.
Default: k = 60; fuse the top 50–100 from each retriever; dedupe by document or chunk id before scoring; send the fused top 100 to a Reranker.
Default k: 60 (Cormack et al. 2009; results stable for k ≈ 20–100)
Combining retriever scores, not ranks: normalise each list then blend, e.g. α·dense + (1−α)·BM25, with α tuned on labelled queries.
Default: Min-max normalise each list per query over its top 100; missing docs take that list's min; α from a 0.0–1.0 sweep in 0.1 steps on the golden set, picked by nDCG@10 with a bootstrap CI.
Labels needed to tune α: 50–200 queries (single global α; per-class α needs ~50 per class)
Turning ranked chunks into the prompt: dedupe, fit a token budget, order, label with source ids, and add the instructions around them.
Default: Budget 3–6k tokens of retrieved context for single-hop Q&A, 8–15k for multi-document synthesis.
Retrieved-context budget: 3–6k tokens (single-hop Q&A; 8–15k for multi-document synthesis)
2023 models used mid-context facts worse than edge ones. In 2026, single facts are found; multi-hop reasoning and aggregation over long context still degrade.
Default: Rerank to 5–10 chunks; order best-first, second-best-last; restate the question after the context block; decompose multi-hop questions into single-hop sub-queries.
Middle-position penalty (2023): > 20 accuracy points (GPT-3.5-Turbo, 20 documents, middle vs first position (Liu et al. 2023))
Making the model answer from supplied sources (retrieved docs, tool results) rather than memory, and checking that it did.
Default: System prompt: answer only from tagged sources, cite ids for every factual sentence, say you cannot find it when sources are insufficient.
Verifier cost: 10–20% of generation cost (small judge model checking 3–8 claims per answer)
Linking each claim in an answer to the exact source passage that supports it, and measuring whether those citations are correct and complete.
Default: Native span citations (Anthropic Citations, or the provider equivalent) when available; else XML-tagged chunks with short ids and an instruction to cite after every factual sentence.
Citation recall target: ≥ 0.9 (customer-facing; measured per sentence on a sample)
The system declining to answer when evidence is missing or confidence is low, trading coverage for accuracy on what it does answer.
Default: Golden set with 10–15% unanswerable items.
Unanswerable share in golden set: 10–15% (enough to measure false-answer rate with ±5-point CI at n ≈ 500)
Reusing a stored answer when a new query's embedding is close enough to a past one. Saves cost and latency; risks returning the wrong answer.
Default: Provider prompt caching first.
Threshold: 0.92–0.97 cosine (start at 0.95; depends on embedding model)