System Design & Interview Loops

A function built from layers of weighted sums and non-linearities whose weights are learned from examples by gradient descent; the machinery under every LLM.

The learned numbers (weights and biases) inside a model. '70B' means 70 billion of them; they set memory and compute cost, not quality on their own.

The architecture behind nearly all modern LLMs: stacked attention and feed-forward blocks that process every token of a sequence in parallel.

The first and most expensive training phase: a model learns language and world knowledge by predicting the next token across trillions of tokens.

The single task an LLM is trained on and runs at inference: given the text so far, assign a probability to every possible next token.

The text, code and examples a model learns from; mix, quality, recency and licensing decide what it knows and how it behaves.

Empirical power laws: loss falls predictably as parameters, data and compute grow, with diminishing returns per extra 10× of compute.

Supervised fine-tuning on (instruction, good response) pairs that turns a base model that continues text into an assistant that follows requests.

Reinforcement learning from human feedback: train a reward model on human preference comparisons, then optimise the LLM to score highly on it.

Methods like DPO that train directly on chosen-vs-rejected answer pairs, getting RLHF-like alignment without a reward model or RL loop.

Low-Rank Adaptation: fine-tune by training small add-on matrices while the original weights stay frozen; cheap, fast, swappable per task or tenant.

A model adapting to a task purely from instructions and examples in its prompt, with no change to its weights.

Including a handful of worked input-output examples in the prompt so the model copies the format, labels and style you want.

Having a model write intermediate reasoning steps before its final answer, which improves accuracy on multi-step problems at a token cost.

Models trained with RL to think at length before answering, trading latency and output tokens for accuracy on maths, code and multi-step work.

A sampling setting that sharpens (low) or flattens (high) next-token probabilities, trading consistency for variety; it does not change what the model knows.

Nucleus sampling: sample only from the smallest set of tokens whose probabilities add up to p (e.g. 0.9), cutting off the unlikely tail.

The raw scores a model outputs for every token in its vocabulary; softmax turns them into next-token probabilities.

Running a trained model to produce output. For LLMs: a compute-bound prefill over the prompt, then a memory-bound decode loop, one token per step.

Stored attention keys and values for past tokens, so each new token skips recomputing the sequence; at long context it fills GPU memory.

Storing weights (and sometimes activations or KV) in 8 or 4 bits instead of 16 to cut memory and speed up inference, at a quality cost you must measure.

Latency from sending a request until the first output token arrives; the main driver of how responsive a streaming LLM app feels.

Running many requests through the GPU together so one weight read serves many tokens: throughput and cost per token improve, per-stream speed falls.

A cheap draft proposes several tokens and the big model verifies them in one pass: same output distribution, fewer sequential big-model steps.

An architecture where each token is routed to a few of many expert sub-networks, so a huge model computes like a much smaller one per token.

Open-weights models publish their parameters for self-hosting; closed models are API-only. The choice trades control and residency for convenience.

The date after which a model's training data ends; it knows nothing newer unless you supply it in the prompt or through tools.

Running an operation twice leaves the same state as running it once. It is the property that makes retries and at-least-once queues safe.

Pushing back on producers when a consumer is saturated: bounded queues, 429s and load shedding instead of unbounded buffering until latency or memory collapses.

A side queue where messages land after N failed attempts, so one poison message stops blocking the pipeline and can be inspected and replayed later.

Wait longer after each failed retry (1 s, 2 s, 4 s…) plus random jitter, so clients don't hammer a struggling service in sync.

Run the new version (green) beside the old (blue), verify it on real infra, then switch all traffic at once, keeping blue warm for instant rollback.

Writing the same change to two stores, e.g. Postgres and a vector index. Done naively it drifts; the outbox pattern plus CDC makes it reliable.

Copy real production requests to a new version whose responses users never see, then compare outputs, latency and cost before risking live traffic.

Release to a small slice of real traffic first (1%, then 5%, 25%…), compare against the old version, and roll back automatically if metrics degrade.

A runtime switch that turns code paths, models or prompts on or off per user, tenant or percentage without a redeploy.

A stable name (e.g. products-read) that points to a physical index, so you can rebuild a new index and repoint readers atomically with no client change.

Background rewriting of an index's segments to drop deleted and overwritten entries, reclaiming space and restoring search speed and recall.

Being able to answer new questions about production from its telemetry. For LLM systems: traces per request with prompts, tool calls, tokens, cost and quality.

Time to live: how long a cached or stored item is valid before it expires. Too long serves stale data; too short throws away savings.

A scheduled job that compares the source of truth with derived stores (index, cache, ACLs) and repairs drift that events and retries missed.

When a dependency fails, serve a reduced but useful response (smaller model, cached answer, sources only) instead of an error page.

A blameless written review after an incident: timeline, impact, root causes, what went well, and owned action items with due dates.

Situation, Task, Action, Result: a structure for behavioural interview answers. Spend most of the time on what you did and what changed, with numbers.

Choosing one option knowing what it costs you on another axis. Senior answers name the axes, the deciding constraint, and when they'd switch.

A senior IC level whose scope spans teams: sets technical direction, de-risks big bets, and multiplies others, mostly through writing and influence.