Neural network
A function built from layers of weighted sums and non-linearities whose weights are learned from examples by gradient descent; the machinery under every LLM.
A neural network is a program that learns a pattern from examples instead of being given rules. It is made of many tiny calculators arranged in layers; each one multiplies its inputs by adjustable numbers called weights, adds them up, and passes the result on.
Training nudges those weights, billions of times, until the network's outputs match the examples. A large language model is a very large neural network trained to guess the next piece of text.
A huge mixing desk with billions of sliders. Nobody sets them by hand; you play it examples, hear how wrong the output is, and a machine nudges every slider a hair in the direction that would have sounded better.
Thinking the network stores facts like rows in a table that can be looked up, edited or deleted. Knowledge is smeared across weights, which is why models hallucinate and why you add facts with Retrieval-Augmented Generation (RAG) rather than surgery on weights.
A feed-forward network computes y = f_L(...f_2(f_1(x))), where each layer is an affine map W·x + b followed by a non-linearity (ReLU, GELU, SwiGLU). Without the non-linearity, stacked layers collapse into one linear map; with it, depth buys expressive power.
Training minimises a loss (for LLMs, cross-entropy on the next Token) with backpropagation, which computes the gradient of the loss for every weight via the chain rule, and an optimiser, AdamW in almost every LLM run, which keeps two running moments per weight. That is why training memory is about 16 bytes per parameter in mixed precision (BF16 weights and gradients, FP32 master copy and two Adam moments) versus 2 bytes to serve in BF16. Forward compute is about 2 × params FLOPs per token; training is about 6 × params per token (forward plus backward). LLMs are one architecture, the Transformer: Attention plus MLP blocks with residual connections and RMSNorm. Knowledge ends up spread across the Parameters & weights, not stored as records, so there is no row to look up, edit or delete.
For LLM work, never train one from scratch: load a pretrained checkpoint with PyTorch 2.x + Hugging Face Transformers. For a small classifier on top of embeddings: 2–3 layer MLP, GELU, AdamW at lr 1e-3, weight decay 0.01, early stopping on validation loss, gradient clipping at 1.0. For tabular data, start with LightGBM or XGBoost; they beat MLPs on most tabular benchmarks.
Clinical intake triage: a 3-layer MLP over 768-dim sentence embeddings (768→256→64→5 classes) has 768×256 + 256 + 256×64 + 64 + 64×5 + 5 = 213,829 weights and trains in under a minute on a laptop CPU over 40,000 labelled notes. The same arithmetic at LLM scale: a 7B model needs 7e9 × 16 B = 112 GB of training state before activations, so a full fine-tune needs at least two 80 GB H100s with FSDP or ZeRO sharding, while serving it in BF16 needs 14 GB.
Default answer when a PM asks 'can we just delete the wrong fact from the model?': no, because knowledge lives in weights, not rows; we change what the model reads (Retrieval-Augmented Generation (RAG)) or retrain behaviour (Fine-tuning). Expect the follow-up 'what about machine unlearning?': it is research-grade, unverifiable for a specific fact, and not something to put in a compliance plan.
Numbers worth knowing
- Training memory
- ~16 bytes/parammixed precision + AdamW, excluding activations
- Serving memory
- 2 bytes/paramBF16 weights, excluding KV cache
- Training compute
- ~6 × params FLOPs/tokenforward + backward pass
- Inference compute
- ~2 × params FLOPs/tokendense model, forward only
How to decide
| When | Do this |
|---|---|
| Tabular data under ~10M rows | Gradient-boosted trees first; an MLP only if trees plateau. |
| Text, images or audio | Fine-tune or prompt a pretrained Transformer; never train from random init. |
| A stakeholder wants one fact removed or changed | Change the retrieved source (Retrieval-Augmented Generation (RAG)) and add an eval case; do not edit weights. |
Cost maths
Training memory ≈ 16 B × params + activations. Full fine-tune of an 8B model: 128 GB of state → 2×H100 80 GB with FSDP, about $6/hour at an assumed $3/GPU-hour (illustrative on-demand price, Oct 2026). The same model with LoRA & parameter-efficient fine-tuning needs ~16 GB of frozen BF16 weights plus a few GB of adapter state → one 24–48 GB GPU at roughly a third of the hourly cost.
Operating it
- OOM at micro-batch 1 → shard optimiser state (FSDP/ZeRO-3) or switch to LoRA
- Validation loss rising while training loss falls for 2+ evals → more data, dropout or earlier stop
- Gradient-norm spikes above 10× the rolling median → lower learning rate, check the data shard
- Train–validation loss gap > 10% for 2 consecutive evals → overfitting alert
- Gradient norm > 10× rolling median, or any NaN loss → page the training owner
- Production accuracy on the weekly labelled sample drops > 3 points from validation
When it goes wrong
A clinical-intake classifier scored 96% on validation and 71% in its first month of production.
A weekly human audit of 200 randomly sampled production notes, compared against the model's labels.
Train and validation were split by row, not by patient; templated notes from the same patient sat on both sides, so the network memorised templates instead of learning the task.
Re-split by patient id, deduplicate near-identical notes, retrain. Honest validation dropped to 84%, matched production within 2 points, and became the number the team reported.
Sources
- Learning representations by back-propagating errors — Rumelhart, Hinton & Williams, 1986paper
- Decoupled Weight Decay Regularization (AdamW) — Loshchilov & Hutter, 2017paper
- Deep Learning — Goodfellow, Bengio & Courville, 2016book
Check yourself
Three questions. Every option is explained once you answer.