A transformer trained to predict the next token over trillions of tokens, then tuned to follow instructions: the engine under chat, agents and RAG.
Default: Hosted frontier model behind an LLM gateway with Model fallback & routing to a second provider; streaming on; temperature 0 for extraction and classification, 0.7 for drafting; max_tokens set per route from measured p99 output + 30%; prompts in git with Prompt versioning; log input, output and reasoning tokens plus stop_reason on every call.
Decode speed, hosted API: 50–150 tokens/s per stream (single request on frontier models, Oct 2026; reasoning models spend hidden tokens before the first visible one)
The unit an LLM reads and writes: a word or word fragment. Context limits, latency and price are all metered in tokens, not characters.
Default: Count with the provider's token-counting endpoint or the model's own tokenizer (tiktoken for OpenAI models, the Hugging Face tokenizer for open weights).
English prose: ≈ 4 chars or 0.75 words per token (100k–200k vocabulary BPE tokenizers)
The maximum number of tokens, prompt plus output, a model can handle in one call. Anything outside it does not exist for the model.
Default: Explicit per-route budget: system + tools ≤ 5k, history capped at 8k with older turns summarised, retrieved context 3–8k, reserved output from measured p99 + 30%.
Frontier input window: 200k–1M+ tokens (hosted APIs, Oct 2026; open-weights models commonly 128k)
The full input sent to an LLM: instructions, examples, retrieved context and the user's message. The main lever for controlling behaviour.
Default: System prompt in git with a version id logged on every request; sections for role, task, rules, output schema; retrieved context in XML-style tags with ids; 3–5 edge-case examples; stable content before one cache breakpoint; provider structured outputs instead of 'please return JSON'; a 200-case golden set run in CI on every prompt change.
Accuracy swing from a one-line instruction edit: ±1–5 points (range seen on 200–500 case golden sets; measure it, never assume zero)
Facts an LLM absorbed into its weights during training: broad but frozen at the cutoff, unattributable, and impossible to update or delete one at a time.
Default: Treat the weights as language and reasoning.
Cutoff-to-release gap: 6–12 months (typical for frontier releases; the model knows nothing after its cutoff)
When an LLM states something fluent and confident that is false or not supported by its sources.
Default: System prompt requires answers only from delimited context with a citation per claim; post-generation check that every cited id exists in the context; abstain when the reranker's top score is below a calibrated threshold; nightly claim-level faithfulness judge on 500 sampled answers; quarterly 200-answer Error analysis.
Faithfulness alert line: < 0.90 claim support (judge-scored daily sample of 500 answers on a grounded route)
Continuing to train a pretrained model on your own examples to change its behaviour, format or style. Not a reliable way to add facts.
Default: Start with LoRA rank 16 on all linear layers of an 8B open-weights model (or the provider's hosted fine-tuning), 2–3 epochs, learning rate ~1e-4, 2,000–10,000 examples, 10% held out.
LoRA trainable share: ≈0.5% (rank 16 on all linear layers of an 8B model (~42M params))
Retrieval-augmented generation: fetch relevant documents for a question, put them in the prompt, and have the LLM answer from them with citations.
Default: Token-based chunks of 400–800 tokens with 10–15% overlap; bge-m3 self-hosted or a hosted embedding model; BM25 plus dense fused with Reciprocal Rank Fusion (RRF) (k = 60); top 50 to a cross-encoder (bge-reranker-v2-m3 or a hosted reranker) keeping 5–8; context with chunk ids and a citation-required prompt; pgvector up to ~10M chunks, Qdrant/OpenSearch beyond; a 200-question golden set run nightly.
Recall@50 target: ≥ 0.90 (on a golden set of real user questions; below this, the reranker and LLM cannot recover)
The full collection of documents a retrieval system can search: the source of truth that RAG answers are built from.
Default: A document registry table (doc_id, source, version, owner, ACL, content_hash, updated_at, deleted_at); CDC or webhook ingestion per source; MinHash dedup at Jaccard ≥ 0.85 on 5-gram shingles; only the latest version marked retrievable; nightly reconciliation; deletion SLA ≤ 1 hour.
Near-duplicate threshold: Jaccard ≥ 0.85 (MinHash on 5-gram word shingles)
The transformer mechanism that lets each token weigh every earlier token in the context when building its representation.
Default: Serve GQA models on vLLM or SGLang with FlashAttention kernels, paged KV, FP8 KV cache where quality holds on your evals, prefix caching on, and max_model_len set to the p99 real context rather than the model maximum.
KV bytes per token: 2 × layers × kv_heads × head_dim × bytes (Llama-3.1-8B BF16 = 128 KB; Llama-3.1-70B BF16 = 320 KB)
Reusing the model's computed state for an identical prompt prefix across requests: cheaper, faster input for repeated context, minus a cache-write premium.
Default: One breakpoint after the static prefix (system + sorted tool schemas + reference docs); a second after conversation history for long agent loops; 5-minute TTL by default, 1-hour only when measured hit rate on it exceeds ~60%; a CI test that asserts the prefix bytes are identical across two renders; log cache_read_input_tokens and cache_creation_input_tokens per request.
Anthropic cache write: 1.25× base input (5-min TTL), 2× (1-hour TTL) (as of Oct 2026)
Constraining generation so output always matches a schema or grammar (e.g. valid JSON) by masking tokens that would break it.
Default: Hosted: strict structured outputs or JSON-schema tools.
Parse failures, free-form JSON: 1–3% (prompt-only JSON on 10k-item extraction runs (illustrative range))
Training a smaller, cheaper student model to imitate a larger teacher model's outputs on your task.
Default: Teacher: the best available frontier model with the production prompt.
DistilBERT trade: 97% of quality, 40% smaller, 60% faster (GLUE, Sanh et al. 2019)
Models or pipelines that handle more than text, such as images, scanned PDFs, audio or video, in the same input or index.
Default: Layout-aware parsing (e.g.
Image token cost (Claude): ⌈w/28⌉ × ⌈h/28⌉ (1092 × 1092 px = 1,521 tokens; capped at 1,568 standard, 4,784 high-resolution (Oct 2026))
Having an LLM translate a natural-language question into a SQL query, run it under a restricted role, and answer from the result.
Default: Semantic layer definitions retrieved with the schema slice; sqlglot parse with table allow-list; read-only role with row-level security; LIMIT 1000; 30 s statement timeout; maximum_bytes_billed or warehouse resource monitor; one retry on execution error; SQL always shown to the user; 150-question golden set scored by execution accuracy.
BIRD human execution accuracy: 92.96% (expert baseline in the BIRD paper; leading systems still below it as of 2026)
RAG where the LLM decides what to retrieve, from which source, and when to search again, in a bounded loop instead of one fixed retrieval step.
Default: Router (small classifier or cheap LLM call) sending ~80% of questions to single-shot RAG; agent built on plain tool calling or LangGraph; max 8 tool calls, 60k-token budget, 45 s timeout; abstain after 2 empty searches; permission-scoped tools; every step traced (OpenTelemetry spans); trajectory evals on 100 multi-hop questions.
Cost multiplier vs single-shot: 3–10× (3–5 tool rounds with context re-sent each round)
Using an LLM with a rubric to grade outputs (faithfulness, relevance, correctness) so evaluation scales beyond human review.
Default: Binary or 3-point rubric per criterion; rationale then verdict in a JSON schema; judge pinned to a dated model version from a different family than the generator; pairwise runs in both orders; 300-item human calibration set with kappa ≥ 0.7; 50 fresh human labels a month to detect judge drift; deterministic checks first.
Strong judge vs human agreement: > 80% (MT-Bench pairwise preferences, Zheng et al. 2023; similar to human-human agreement)
A function built from layers of weighted sums and non-linearities whose weights are learned from examples by gradient descent; the machinery under every LLM.
Default: For LLM work, never train one from scratch: load a pretrained checkpoint with PyTorch 2.x + Hugging Face Transformers.
Training memory: ~16 bytes/param (mixed precision + AdamW, excluding activations)
The learned numbers (weights and biases) inside a model. '70B' means 70 billion of them; they set memory and compute cost, not quality on their own.
Default: Size every deployment as weights + KV(p95 context × target concurrency) + 10% overhead, and keep at least 30% of GPU memory for KV.
70B weights: 140 GB BF16 / 70 GB FP8 / ~38 GB INT4 (excluding KV and overhead)
The architecture behind nearly all modern LLMs: stacked attention and feed-forward blocks that process every token of a sequence in parallel.
Default: Do not modify the architecture; choose a checkpoint.
Llama-3.1-8B shape: 32 layers · d=4,096 · 32 Q / 8 KV heads · vocab 128,256 (public config)
The first and most expensive training phase: a model learns language and world knowledge by predicting the next token across trillions of tokens.
Default: Do not pretrain.
Pretraining compute: ≈ 6 × params × tokens FLOPs (dense transformer)
The single task an LLM is trained on and runs at inference: given the text so far, assign a probability to every possible next token.
Default: Always set max_tokens explicitly per route, always check the stop/finish reason ('length' or 'max_tokens' means truncated), stream output to users, and use Structured output / decoding for anything a program parses.
Forward passes per answer: 1 per output token (plus one prefill for the prompt)
The text, code and examples a model learns from; mix, quality, recency and licensing decide what it knows and how it behaves.
Default: For fine-tuning: 1k–10k examples sampled from real traffic, MinHash near-dedup, PII redaction, expert-corrected outputs, split by source document or customer (never by row), and a 13-gram overlap check against every eval set before training.
FineWeb: 15T tokens from 96 Common Crawl snapshots (Penedo et al., 2024)
Empirical power laws: loss falls predictably as parameters, data and compute grow, with diminishing returns per extra 10× of compute.
Default: For product decisions, ignore the training curve and run your own scaling sweep: 2–3 model sizes × 2–3 reasoning efforts on your Golden set, plotted as accuracy vs cost per 1k requests.
Compute-optimal ratio: ~20 tokens per parameter (Chinchilla, fixed training budget)
Supervised fine-tuning on (instruction, good response) pairs that turns a base model that continues text into an assistant that follows requests.
Default: LoRA (rank 16, alpha 32, all linear layers) on the vendor's instruct checkpoint; loss on assistant tokens only; the tokenizer's own apply_chat_template for both training and serving; lr 1e-4, 2–3 epochs, cosine decay; 10% held out by source document; compare against a few-shot prompt baseline before shipping.
LIMA: 1,000 examples (enough for strong format/alignment on a 65B base)
Reinforcement learning from human feedback: train a reward model on human preference comparisons, then optimise the LLM to score highly on it.
Default: Product teams: do not run PPO.
InstructGPT result: 1.3B RLHF preferred over 175B GPT-3 (labeller preference, Ouyang et al. 2022)
Methods like DPO that train directly on chosen-vs-rejected answer pairs, getting RLHF-like alignment without a reward model or RL loop.
Default: DPO with LoRA (TRL or similar), β = 0.1, lr 5e-6, 1–2 epochs, reference log-probs precomputed, pairs built from real user or expert edits, length-balanced; evaluate with a length-controlled Pairwise comparison plus a general-capability regression set.
DPO β: 0.1 default (0.05–0.5) (higher = stay closer to reference)
Low-Rank Adaptation: fine-tune by training small add-on matrices while the original weights stay frozen; cheap, fast, swappable per task or tenant.
Default: Rank 16, alpha 32, dropout 0.05, all linear layers, lr 1e-4–2e-4, 2–3 epochs, BF16 base (QLoRA 4-bit only when memory forces it).
Trainable params: ~42M (0.5%) (Llama-3.1-8B, r = 16, all linear layers)
A model adapting to a task purely from instructions and examples in its prompt, with no change to its weights.
Default: Start every new task with instructions + 3–8 examples, the stable part placed first and marked for Prompt caching, the current date and variable data last.
Cached input price: 0.1× base input (Anthropic cache reads; writes 1.25× (5-min TTL) or 2× (1-hour))
Including a handful of worked input-output examples in the prompt so the model copies the format, labels and style you want.
Default: 3–6 examples covering every label (at least one per rare label), one borderline case, consistent format, placed in the cached prefix; dynamic retrieval of k = 4–8 examples from a labelled bank when inputs are heterogeneous; Structured output / decoding for machine-read output.
Useful example count: 3–8 (gains usually flatten after ~8 on classification)
Having a model write intermediate reasoning steps before its final answer, which improves accuracy on multi-step problems at a token cost.
Default: Non-reasoning models: ask for reasoning in a separate field (e.g.
GSM8K, PaLM 540B: 17.9% → 56.9% (standard vs CoT prompting, Wei et al. 2022)
Models trained with RL to think at length before answering, trading latency and output tokens for accuracy on maths, code and multi-step work.
Default: Route by difficulty: non-thinking (or lowest effort) for extraction, classification and chat; medium effort for multi-step analysis; high only for the hardest class, verified on your eval.
R1-Zero AIME 2024 pass@1: 15.6% → 71.0% (RL alone, DeepSeek-R1 paper)
A sampling setting that sharpens (low) or flattens (high) next-token probabilities, trading consistency for variety; it does not change what the model knows.
Default: Extraction, classification, SQL and code: 0–0.2.
Distinct outputs at T = 0: 80 of 1,000 (Qwen3-235B, Thinking Machines 2025, standard kernels)
Nucleus sampling: sample only from the smallest set of tokens whose probabilities add up to p (e.g. 0.9), cutting off the unlikely tail.
Default: top_p = 1.0 for extraction and code at low temperature; 0.95 for open-ended generation; min-p 0.05 on self-hosted stacks when running temperature > 1.
Standard nucleus: p = 0.9–0.95 (open-ended generation)
The raw scores a model outputs for every token in its vocabulary; softmax turns them into next-token probabilities.
Default: Use logprobs only where the provider exposes them; fit any confidence threshold on ≥ 1,000 labelled held-out items; track expected calibration error (ECE) per model version; prefer label tokens that are single tokens in the model's tokenizer.
Vocabulary (logits per step): 128,256 (Llama 3 tokenizer)
Running a trained model to produce output. For LLMs: a compute-bound prefill over the prompt, then a memory-bound decode loop, one token per step.
Default: vLLM or SGLang, FP8 weights, PagedAttention (default), chunked prefill on with max_num_batched_tokens 4k–8k, prefix caching on, max_model_len = p99 real context, streaming to clients.
Decode floor, 70B FP8 on 2×H100: ~10.4 ms/token (~96 tok/s per stream) (70 GB ÷ 6.7 TB/s, before KV reads)
Stored attention keys and values for past tokens, so each new token skips recomputing the sequence; at long context it fills GPU memory.
Default: vLLM or SGLang, paged KV, FP8 KV cache, prefix caching on, max_model_len set to the p99 real context (not the model maximum), gpu_memory_utilization 0.9, and a separate pool or tier for contexts above 32k.
Llama-3.1-70B KV: 320 KB/token BF16, 160 KB FP8 (80 layers × 8 KV heads × 128 dim)
Storing weights (and sometimes activations or KV) in 8 or 4 bits instead of 16 to cut memory and speed up inference, at a quality cost you must measure.
Default: FP8 weights + FP8 KV on Hopper-class GPUs; AWQ INT4 (group size 128) only where memory forces it and the sliced eval passes; never ship a quantized artifact without re-running the Golden set split by task type (numeric, code, multilingual, long-context).
70B weights: 140 → 70 → ~38 GB (BF16 → FP8 → INT4)
Latency from sending a request until the first output token arrives; the main driver of how responsive a streaming LLM app feels.
Default: Stream every user-facing response; static instructions and documents first in the prompt with a cache breakpoint; retrieval and reranking in parallel where possible; client-side TTFT measurement (RUM) alongside server metrics; SLO of p95 TTFT < 1 s for chat and < 2.5 s for RAG.
Chat TTFT target: p95 < 1 s (interactive assistants)
Running many requests through the GPU together so one weight read serves many tokens: throughput and cost per token improve, per-stream speed falls.
Default: Continuous batching (default in vLLM/SGLang), chunked prefill on, max_num_seqs set from a load test at real token distributions, ITL p95 and TTFT p95 SLOs as the stopping rule, 30% headroom; all non-interactive jobs through provider batch APIs (~50% discount).
Throughput gain, batch 1 → 32: ~20× (91 → 1,778 tok/s) (8B on 1×H100, this page's load test)
A cheap draft proposes several tokens and the big model verifies them in one pass: same output distribution, fewer sequential big-model steps.
Default: Enable on low-concurrency, latency-sensitive tiers only: prompt-lookup (n-gram) for code edits and RAG quoting, EAGLE-style drafters or built-in MTP heads for general chat; k = 3–5; measure acceptance rate and tokens/s per GPU in an A/B before rollout; disable on saturated batch pools.
Leviathan et al. speed-up: 2–3× (T5-XXL, identical outputs)
An architecture where each token is routed to a few of many expert sub-networks, so a huge model computes like a much smaller one per token.
Default: Self-host MoE only with steady, high-batch traffic: vLLM or SGLang with expert parallelism, FP8 (or the model's native MXFP4/FP8) weights, expert load balancing enabled, and per-rank step-time monitoring.
DeepSeek-V3: 671B total / 37B active (1 shared + 8 of 256 routed experts)
Open-weights models publish their parameters for self-hosting; closed models are API-only. The choice trades control and residency for convenience.
Default: Start on a closed API behind an LLM gateway.
Llama licence MAU threshold: 700M monthly active users (Llama 3.x Community Licence)
The date after which a model's training data ends; it knows nothing newer unless you supply it in the prompt or through tools.
Default: Today's date and key current facts (product versions, policy effective dates) in every system prompt; retrieval or search for any time-sensitive question with effective dates in the index; model cutoff recorded in the model registry; ≥ 20 post-cutoff questions in the Golden set that the model must answer from retrieval or abstain on.
Release lag: 3–12 months after cutoff (typical across major providers)