Transformer
The architecture behind nearly all modern LLMs: stacked attention and feed-forward blocks that process every token of a sequence in parallel.
A transformer is the design inside almost every modern Large Language Model (LLM). Text is split into tokens, each token becomes a list of numbers, and those numbers pass up through a stack of identical layers.
In each layer, Attention lets every token look at the tokens before it to gather context (so 'bank' can tell whether it means a river or money), then a small Neural network refines each token on its own. At the top, the model outputs a score for every possible next token.
A meeting where, in every round, each participant listens to whoever is most relevant to them and then updates their own notes. After dozens of rounds, everyone's notes reflect the whole conversation.
Thinking the transformer reads text left to right like a person during training. It processes every position in parallel; only generation is sequential, one token at a time.
Tokens are embedded and given positions with rotary embeddings (RoPE). Each block applies multi-head self-Attention (softmax(QKᵀ/√d)·V per head) and a SwiGLU MLP, each wrapped in a residual connection with pre-norm RMSNorm. LLMs are decoder-only with a causal mask: position i attends only to positions ≤ i, so Next-token prediction trains on every position of a sequence in one parallel pass.
Concrete shape, Llama-3.1-8B: 32 layers, hidden size 4,096, 32 query heads sharing 8 key-value heads (grouped-query attention, which cuts the KV cache 4×), MLP width 14,336, vocabulary 128,256. About 80% of the per-layer weights sit in the MLP. Attention-score compute grows with context: at ~27k tokens a decode step's QKᵀ and AV work equals all of its weight matmuls, and in prefill the total grows quadratically. FlashAttention removes the quadratic memory by tiling; the quadratic compute remains, which is what sliding-window, hybrid linear-attention and latent-attention (MLA) designs attack. Encoder-only transformers (BERT-style) still power encoders for embeddings and rerankers.
Do not modify the architecture; choose a checkpoint. Prefer GQA or MLA models for serving (smaller KV cache), serve with vLLM or SGLang using FlashAttention kernels, and copy rope_scaling and max_position_embeddings from the model card exactly.
A 2,000-token internal-wiki page goes through all 32 layers of an 8B model in one prefill pass (2 × 8e9 × 2,000 ≈ 3.2e13 FLOPs, under 0.1 s on one H100 at realistic utilisation). Each generated token then runs one decode step through all 32 layers, attending to the 2,000 cached keys and values per layer instead of recomputing them. In the attention heatmap for 'river bank', the head attending from 'bank' puts most of its weight on 'river' — the disambiguation happens there.
If asked to sketch a block, draw: RMSNorm → attention (with GQA) → residual add → RMSNorm → SwiGLU MLP → residual add, repeated N times, and say where the KV cache lives. The follow-up is 'why did it replace RNNs?': parallel training across positions and direct long-range paths; the price is attention cost that grows with context, which is why long-context serving is a memory problem.
| The | river | bank | was | muddy | |
|---|---|---|---|---|---|
| The | 1.00 | · | · | · | · |
| river | 0.30 | 0.70 | · | · | · |
| bank | 0.10 | 0.62 | 0.28 | · | · |
| was | 0.05 | 0.20 | 0.60 | 0.15 | · |
| muddy | 0.05 | 0.30 | 0.45 | 0.10 | 0.10 |
Numbers worth knowing
- Llama-3.1-8B shape
- 32 layers · d=4,096 · 32 Q / 8 KV heads · vocab 128,256public config
- Context where attention FLOPs = weight FLOPs
- ~27k tokensLlama-3.1-8B decode step
- KV reduction from GQA
- 4×32 query heads sharing 8 KV heads
How to decide
| When | Do this |
|---|---|
| Generating text | Decoder-only LLM. |
| Embeddings or reranking at high QPS | Encoder-only model (Bi-encoder or Cross-encoder); 10–100× cheaper per item. |
| Context routinely > 100k tokens | Prefer models with GQA/MLA or hybrid attention and test retrieval at that length before committing. |
Cost maths
Prefill FLOPs ≈ 2 × params × prompt tokens + attention term ≈ 4 × layers × d × n² / 2. For an 8B model at n = 2k the weight term (3.2e13) dominates; at n = 128k the attention term (~4.3e15, before GQA savings on memory) exceeds the weight term (~2e15). Long prompts cost superlinearly in compute; price long-context tiers accordingly.
Operating it
- Median prompt length crosses ~30k tokens → attention dominates; evaluate long-context-efficient architectures
- KV cache, not weights, is > 50% of GPU memory at p95 → prefer GQA/MLA models
- Needle-in-a-haystack recall at your p95 length < 90% → do not rely on raw context; use retrieval
- Quality by prompt-length bucket (4k/16k/64k) — alert if any bucket drops > 5 points vs baseline
- Prefill time per 1k tokens p95 — alert on > 30% growth after a deploy
- Repetition rate (repeated 4-grams per answer) > 2× baseline
When it goes wrong
After a checkpoint conversion, answers to questions over 20k-token legal briefs turned into repetitive gibberish; short prompts were fine.
A long-context regression eval (questions at 4k, 16k, 32k) dropped from 88% to 9% at 16k+ while the 4k bucket was unchanged.
The conversion script dropped rope_scaling from config.json, so positions beyond the original 8k training length were encoded incorrectly.
Restored the config, added a config-diff check against the upstream model card to the deploy pipeline, and kept the length-bucketed eval as a release gate.
Related terms
Sources
- Attention Is All You Need — Vaswani et al., 2017paper
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., 2023paper
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Dao et al., 2022paper
- RoFormer: Enhanced Transformer with Rotary Position Embedding — Su et al., 2021paper
Check yourself
Three questions. Every option is explained once you answer.