One line per term, the default to reach for, and the number to remember. Print it or save it as a PDF.
A function built from layers of weighted sums and non-linearities whose weights are learned from examples by gradient descent; the machinery under every LLM.
Default: For LLM work, never train one from scratch: load a pretrained checkpoint with PyTorch 2.x + Hugging Face Transformers.
Training memory: ~16 bytes/param (mixed precision + AdamW, excluding activations)
The learned numbers (weights and biases) inside a model. '70B' means 70 billion of them; they set memory and compute cost, not quality on their own.
Default: Size every deployment as weights + KV(p95 context × target concurrency) + 10% overhead, and keep at least 30% of GPU memory for KV.
70B weights: 140 GB BF16 / 70 GB FP8 / ~38 GB INT4 (excluding KV and overhead)
The architecture behind nearly all modern LLMs: stacked attention and feed-forward blocks that process every token of a sequence in parallel.
Default: Do not modify the architecture; choose a checkpoint.
Llama-3.1-8B shape: 32 layers · d=4,096 · 32 Q / 8 KV heads · vocab 128,256 (public config)
The first and most expensive training phase: a model learns language and world knowledge by predicting the next token across trillions of tokens.
Default: Do not pretrain.
Pretraining compute: ≈ 6 × params × tokens FLOPs (dense transformer)
The single task an LLM is trained on and runs at inference: given the text so far, assign a probability to every possible next token.
Default: Always set max_tokens explicitly per route, always check the stop/finish reason ('length' or 'max_tokens' means truncated), stream output to users, and use Structured output / decoding for anything a program parses.
Forward passes per answer: 1 per output token (plus one prefill for the prompt)
The text, code and examples a model learns from; mix, quality, recency and licensing decide what it knows and how it behaves.
Default: For fine-tuning: 1k–10k examples sampled from real traffic, MinHash near-dedup, PII redaction, expert-corrected outputs, split by source document or customer (never by row), and a 13-gram overlap check against every eval set before training.
FineWeb: 15T tokens from 96 Common Crawl snapshots (Penedo et al., 2024)
Empirical power laws: loss falls predictably as parameters, data and compute grow, with diminishing returns per extra 10× of compute.
Default: For product decisions, ignore the training curve and run your own scaling sweep: 2–3 model sizes × 2–3 reasoning efforts on your Golden set, plotted as accuracy vs cost per 1k requests.
Compute-optimal ratio: ~20 tokens per parameter (Chinchilla, fixed training budget)
Supervised fine-tuning on (instruction, good response) pairs that turns a base model that continues text into an assistant that follows requests.
Default: LoRA (rank 16, alpha 32, all linear layers) on the vendor's instruct checkpoint; loss on assistant tokens only; the tokenizer's own apply_chat_template for both training and serving; lr 1e-4, 2–3 epochs, cosine decay; 10% held out by source document; compare against a few-shot prompt baseline before shipping.
LIMA: 1,000 examples (enough for strong format/alignment on a 65B base)
Reinforcement learning from human feedback: train a reward model on human preference comparisons, then optimise the LLM to score highly on it.
Default: Product teams: do not run PPO.
InstructGPT result: 1.3B RLHF preferred over 175B GPT-3 (labeller preference, Ouyang et al. 2022)
Methods like DPO that train directly on chosen-vs-rejected answer pairs, getting RLHF-like alignment without a reward model or RL loop.
Default: DPO with LoRA (TRL or similar), β = 0.1, lr 5e-6, 1–2 epochs, reference log-probs precomputed, pairs built from real user or expert edits, length-balanced; evaluate with a length-controlled Pairwise comparison plus a general-capability regression set.
DPO β: 0.1 default (0.05–0.5) (higher = stay closer to reference)
Low-Rank Adaptation: fine-tune by training small add-on matrices while the original weights stay frozen; cheap, fast, swappable per task or tenant.
Default: Rank 16, alpha 32, dropout 0.05, all linear layers, lr 1e-4–2e-4, 2–3 epochs, BF16 base (QLoRA 4-bit only when memory forces it).
Trainable params: ~42M (0.5%) (Llama-3.1-8B, r = 16, all linear layers)
A model adapting to a task purely from instructions and examples in its prompt, with no change to its weights.
Default: Start every new task with instructions + 3–8 examples, the stable part placed first and marked for Prompt caching, the current date and variable data last.
Cached input price: 0.1× base input (Anthropic cache reads; writes 1.25× (5-min TTL) or 2× (1-hour))
Including a handful of worked input-output examples in the prompt so the model copies the format, labels and style you want.
Default: 3–6 examples covering every label (at least one per rare label), one borderline case, consistent format, placed in the cached prefix; dynamic retrieval of k = 4–8 examples from a labelled bank when inputs are heterogeneous; Structured output / decoding for machine-read output.
Useful example count: 3–8 (gains usually flatten after ~8 on classification)
Having a model write intermediate reasoning steps before its final answer, which improves accuracy on multi-step problems at a token cost.
Default: Non-reasoning models: ask for reasoning in a separate field (e.g.
GSM8K, PaLM 540B: 17.9% → 56.9% (standard vs CoT prompting, Wei et al. 2022)
Models trained with RL to think at length before answering, trading latency and output tokens for accuracy on maths, code and multi-step work.
Default: Route by difficulty: non-thinking (or lowest effort) for extraction, classification and chat; medium effort for multi-step analysis; high only for the hardest class, verified on your eval.
R1-Zero AIME 2024 pass@1: 15.6% → 71.0% (RL alone, DeepSeek-R1 paper)
A sampling setting that sharpens (low) or flattens (high) next-token probabilities, trading consistency for variety; it does not change what the model knows.
Default: Extraction, classification, SQL and code: 0–0.2.
Distinct outputs at T = 0: 80 of 1,000 (Qwen3-235B, Thinking Machines 2025, standard kernels)
Nucleus sampling: sample only from the smallest set of tokens whose probabilities add up to p (e.g. 0.9), cutting off the unlikely tail.
Default: top_p = 1.0 for extraction and code at low temperature; 0.95 for open-ended generation; min-p 0.05 on self-hosted stacks when running temperature > 1.
Standard nucleus: p = 0.9–0.95 (open-ended generation)
The raw scores a model outputs for every token in its vocabulary; softmax turns them into next-token probabilities.
Default: Use logprobs only where the provider exposes them; fit any confidence threshold on ≥ 1,000 labelled held-out items; track expected calibration error (ECE) per model version; prefer label tokens that are single tokens in the model's tokenizer.
Vocabulary (logits per step): 128,256 (Llama 3 tokenizer)
Running a trained model to produce output. For LLMs: a compute-bound prefill over the prompt, then a memory-bound decode loop, one token per step.
Default: vLLM or SGLang, FP8 weights, PagedAttention (default), chunked prefill on with max_num_batched_tokens 4k–8k, prefix caching on, max_model_len = p99 real context, streaming to clients.
Decode floor, 70B FP8 on 2×H100: ~10.4 ms/token (~96 tok/s per stream) (70 GB ÷ 6.7 TB/s, before KV reads)
Stored attention keys and values for past tokens, so each new token skips recomputing the sequence; at long context it fills GPU memory.
Default: vLLM or SGLang, paged KV, FP8 KV cache, prefix caching on, max_model_len set to the p99 real context (not the model maximum), gpu_memory_utilization 0.9, and a separate pool or tier for contexts above 32k.
Llama-3.1-70B KV: 320 KB/token BF16, 160 KB FP8 (80 layers × 8 KV heads × 128 dim)
Storing weights (and sometimes activations or KV) in 8 or 4 bits instead of 16 to cut memory and speed up inference, at a quality cost you must measure.
Default: FP8 weights + FP8 KV on Hopper-class GPUs; AWQ INT4 (group size 128) only where memory forces it and the sliced eval passes; never ship a quantized artifact without re-running the Golden set split by task type (numeric, code, multilingual, long-context).
70B weights: 140 → 70 → ~38 GB (BF16 → FP8 → INT4)
Latency from sending a request until the first output token arrives; the main driver of how responsive a streaming LLM app feels.
Default: Stream every user-facing response; static instructions and documents first in the prompt with a cache breakpoint; retrieval and reranking in parallel where possible; client-side TTFT measurement (RUM) alongside server metrics; SLO of p95 TTFT < 1 s for chat and < 2.5 s for RAG.
Chat TTFT target: p95 < 1 s (interactive assistants)
Running many requests through the GPU together so one weight read serves many tokens: throughput and cost per token improve, per-stream speed falls.
Default: Continuous batching (default in vLLM/SGLang), chunked prefill on, max_num_seqs set from a load test at real token distributions, ITL p95 and TTFT p95 SLOs as the stopping rule, 30% headroom; all non-interactive jobs through provider batch APIs (~50% discount).
Throughput gain, batch 1 → 32: ~20× (91 → 1,778 tok/s) (8B on 1×H100, this page's load test)
A cheap draft proposes several tokens and the big model verifies them in one pass: same output distribution, fewer sequential big-model steps.
Default: Enable on low-concurrency, latency-sensitive tiers only: prompt-lookup (n-gram) for code edits and RAG quoting, EAGLE-style drafters or built-in MTP heads for general chat; k = 3–5; measure acceptance rate and tokens/s per GPU in an A/B before rollout; disable on saturated batch pools.
Leviathan et al. speed-up: 2–3× (T5-XXL, identical outputs)
An architecture where each token is routed to a few of many expert sub-networks, so a huge model computes like a much smaller one per token.
Default: Self-host MoE only with steady, high-batch traffic: vLLM or SGLang with expert parallelism, FP8 (or the model's native MXFP4/FP8) weights, expert load balancing enabled, and per-rank step-time monitoring.
DeepSeek-V3: 671B total / 37B active (1 shared + 8 of 256 routed experts)
Open-weights models publish their parameters for self-hosting; closed models are API-only. The choice trades control and residency for convenience.
Default: Start on a closed API behind an LLM gateway.
Llama licence MAU threshold: 700M monthly active users (Llama 3.x Community Licence)
The date after which a model's training data ends; it knows nothing newer unless you supply it in the prompt or through tools.
Default: Today's date and key current facts (product versions, policy effective dates) in every system prompt; retrieval or search for any time-sensitive question with effective dates in the index; model cutoff recorded in the model registry; ≥ 20 post-cutoff questions in the Golden set that the model must answer from retrieval or abstain on.
Release lag: 3–12 months after cutoff (typical across major providers)
Running an operation twice leaves the same state as running it once. It is the property that makes retries and at-least-once queues safe.
Default: Caller-minted Idempotency-Key (UUIDv7) on every mutating API; Postgres table idempotency_keys(account_id, key, request_hash, response, created_at) with a unique index on (account_id, key), written in the same transaction as the side effect; 24 h retention for synchronous APIs, retention ≥ queue retention for consumers.
Stripe idempotency-key retention: 24 hours (keys can be pruned after that; a retry later is treated as new)
Pushing back on producers when a consumer is saturated: bounded queues, 429s and load shedding instead of unbounded buffering until latency or memory collapses.
Default: Gateway token bucket per provider at 90% of the published TPM/RPM limit; bounded queue per dependency (size = Little's-law concurrency × 1.5); max queue wait 10 s for interactive, minutes for batch; 429 + Retry-After on overflow; priority classes interactive > batch > eval; deadlines propagated via request headers.
Latency multiplier at 90% utilisation: 10× service time (M/M/1 approximation; real tails are worse with bursty LLM traffic)
A side queue where messages land after N failed attempts, so one poison message stops blocking the pipeline and can be inspected and replayed later.
Default: SQS redrive policy with maxReceiveCount=4, DLQ retention 14 days (source 4 days), CloudWatch alarm on ApproximateNumberOfMessagesVisible >0 for customer-facing queues; for Kafka, a <topic>.dlq topic with error/attempt/offset headers; error classifier that sends schema/validation errors straight to the DLQ.
maxReceiveCount: 3–5 (with exponential backoff between attempts; higher only for known-transient dependencies)
Wait longer after each failed retry (1 s, 2 s, 4 s…) plus random jitter, so clients don't hammer a struggling service in sync.
Default: Full jitter, base 1 s, cap 30 s, max 4 retries, bounded by the request deadline; honour Retry-After; retry at the SDK layer only (disable UI and gateway retries); circuit breaker opens after 50% errors over 20 requests; retry budget 10% of requests.
Calls from layered retries: 3 × 3 × 3 = 27 (three layers each retrying 3 times for one user action)
Run the new version (green) beside the old (blue), verify it on real infra, then switch all traffic at once, keeping blue warm for instant rollback.
Default: Switch via load-balancer target groups (ALB weighted), Kubernetes Service selector, Argo Rollouts blueGreen strategy, or an Elasticsearch/OpenSearch/Qdrant alias; dual-write during overlap; automated smoke + golden-set check before flip; blue kept warm 24 h for stateless, 7 days for indexes.
Infra during overlap: 2× serving cost (both colours at full capacity; vector indexes also 2× storage)
Writing the same change to two stores, e.g. Postgres and a vector index. Done naively it drifts; the outbox pattern plus CDC makes it reliable.
Default: Postgres transactional outbox; Debezium Postgres connector (pgoutput) → Kafka topic keyed by entity id; idempotent upsert consumer with version check (WHERE incoming_version > stored_version); nightly reconciliation by content hash; outbox rows deleted after 7 days.
Typical CDC end-to-end lag: 1–5 s (Debezium + Kafka at moderate load, excluding embedding time)
Copy real production requests to a new version whose responses users never see, then compare outputs, latency and cost before risking live traffic.
Default: Gateway-level async mirror at 5% (Envoy/Istio mirrorPercentage or app fork), separate API key and concurrency pool, side-effecting tools stubbed with recorded responses, outputs logged with the primary's to a comparison table, nightly pairwise judge run with position randomisation, 7–14 day window.
Sample rate: 1–10% (enough for ≥1,000 paired comparisons per key intent)
Release to a small slice of real traffic first (1%, then 5%, 25%…), compare against the old version, and roll back automatically if metrics degrade.
Default: Flag-driven split by hashed user/tenant id; concurrent control slice of equal size; steps 1/5/25/50/100; gates on 5xx, p95 latency, refusal rate, citation validity and thumbs-down; automated rollback (Argo Rollouts/Flagger analysis or flag guardrails); bake ≥1 h at 1–5%, ≥24 h before 100% for prompt or model changes.
Samples at 1% of 50k req/day: 500/day (detects a 5xx spike; cannot detect a 0.2% quality regression)
A runtime switch that turns code paths, models or prompts on or off per user, tenant or percentage without a redeploy.
Default: LaunchDarkly, Unleash, Flagsmith or Statsig behind the OpenFeature SDK; local evaluation with streamed rules; safe default coded for every flag; separate flags for model id, prompt version and tool enablement; bucketing by tenant id for B2B; 90-day expiry lint on release flags; flag changes logged to the audit trail.
Flag evaluation latency: <1 ms (local) (SDK evaluates cached rules in-process; remote evaluation adds a network hop)
A stable name (e.g. products-read) that points to a physical index, so you can rebuild a new index and repoint readers atomically with no client change.
Default: Read alias + write alias per logical index; physical names encode embedding model, dimension, chunker and version; swaps via Elasticsearch/OpenSearch _aliases or Qdrant change_aliases; config-row emulation for pgvector; old index retained 7 days then deleted by a scheduled job.
Alias swap time: <1 s (single atomic cluster-state update in Elasticsearch/OpenSearch)
Background rewriting of an index's segments to drop deleted and overwritten entries, reclaiming space and restoring search speed and recall.
Default: Leave engine optimizers on (Qdrant deleted_threshold 0.2, Milvus auto-compaction, Elasticsearch tiered merge); per-table autovacuum tuning for pgvector (vacuum_scale_factor 0.02–0.05); heavy force-merge or REINDEX only off-peak and only on cold indexes; rebuild + alias swap when weekly churn >30%.
Qdrant vacuum trigger: 20% deleted (default optimizer `deleted_threshold` per segment)
Being able to answer new questions about production from its telemetry. For LLM systems: traces per request with prompts, tool calls, tokens, cost and quality.
Default: OpenTelemetry SDK with GenAI semantic conventions (pinned version); OTLP to Langfuse, Phoenix, Honeycomb or Datadog; spans for retrieval, rerank, generation, each tool; attributes for model, prompt version, tokens, cost, tenant id; content capture redacted, 100% of errors and negative feedback + 5% sample; content retention 30 days, metrics 13 months.
Content capture rate: 100% errors/negative feedback, 5–10% otherwise (keeps storage and PII exposure bounded)
Time to live: how long a cached or stored item is valid before it expires. Too long serves stale data; too short throws away savings.
Default: Redis with SET key value EX <ttl> and ±10% jitter; cache keys include corpus/prompt version; CDC-driven invalidation by tag; single-flight on misses; DynamoDB TTL with an expires_at filter in reads; retention TTLs enforced in the store (S3 lifecycle rules, index ILM).
Anthropic prompt cache TTL: 5 min default, 1 h option (1-hour writes cost 2× base input vs 1.25× for 5-minute; reads 0.1×)
A scheduled job that compares the source of truth with derived stores (index, cache, ACLs) and repairs drift that events and retries missed.
Default: Nightly job (Temporal workflow or Kubernetes CronJob) comparing registry vs index on (doc_id, content_hash, acl_hash, model_version); 1,024 hash buckets; repairs enqueued to the normal ingestion queue at a capped rate; delete circuit breaker at 1% of corpus; ACL-only sync every 15 minutes for permissioned sources; report emitted as metrics.
Delete circuit breaker: abort if >1% would be deleted (guards against empty or truncated source reads)
When a dependency fails, serve a reduced but useful response (smaller model, cached answer, sources only) instead of an error page.
Default: Ladder defined per feature in config; circuit breaker per provider (open at 50% errors over 20 calls or p95 >2× SLO); fallback provider through an LLM gateway (LiteLLM, Portkey, or in-house); 1–5% of traffic routed to the fallback continuously to keep it warm and evaluated; rung recorded on traces; user-visible degraded banner.
Keep-warm traffic to fallback: 1–5% (proves the fallback works and keeps its eval current)
A blameless written review after an incident: timeline, impact, root causes, what went well, and owned action items with due dates.
Default: Shared template (summary, impact, timeline, contributing factors, went well, actions); triggers written into the on-call policy; draft within 5 business days; review meeting; actions as tickets with P0/P1/P2 and owners; monthly audit of open actions; quality incidents (wrong answers) count as incidents.
Draft deadline: ≤5 business days (memory and logs fade fast)
Situation, Task, Action, Result: a structure for behavioural interview answers. Spend most of the time on what you did and what changed, with numbers.
Default: 6–8 prepared stories, each 2–3 minutes; ~15/10/55/20 split across S/T/A/R; first-person actions; result with baseline → after, sample size and timeframe; one dissenting stakeholder; one 'would do differently'; each story tagged to the prompts and leadership principles it answers.
Answer length: 2–3 minutes (leaves time for follow-ups in a 45-minute loop with 3–4 questions)
Choosing one option knowing what it costs you on another axis. Senior answers name the axes, the deciding constraint, and when they'd switch.
Default: For any significant choice: a short options × axes table with numbers, the binding constraint, the switch condition, and a one-way/two-way door label; ADR for one-way doors; two-way doors decided by the owning engineer within days.
Analysis for a two-way door: hours to days (cheap to reverse; bias to action)
A senior IC level whose scope spans teams: sets technical direction, de-risks big bets, and multiplies others, mostly through writing and influence.
Default: In interviews: open with org scope (teams, owners), then build vs buy, rollout plan across teams, success metrics with numbers, risks and how you'd de-risk them, and what you personally do.
Teams influenced: typically 2+ teams (staff scope; principal is usually org-wide)