Backpropagation is how a neural network works out how to improve. After a forward pass produces a prediction and a loss, backpropagation applies the chain rule of calculus backwards through the network to compute the gradient of the loss with respect to every weight. An optimiser then nudges each weight against its gradient. Repeated millions of times, this is how every modern neural network, including large language models, is trained.

What problem does backpropagation solve?

A neural network can have thousands, millions or billions of weights. To reduce its error, you need to know how much each weight contributed to that error. Testing each weight separately (nudge it, rerun the network, measure the change) would need one full pass per weight, which is impossible at scale. Backpropagation computes all of those gradients in a single backward pass that costs roughly as much as the forward pass. That efficiency, popularised for neural networks in a 1986 paper by Rumelhart, Hinton and Williams, is what makes deep learning practical.

What happens in the forward pass?

Each layer takes the previous layer's outputs, computes weighted sums plus biases, and applies an activation function such as ReLU (max(0, z)) or sigmoid. The final layer produces a prediction, and a loss function compares it with the true answer: squared error for numbers, cross-entropy for classes. During the forward pass, the network stores the intermediate values, because the backward pass needs them.

What is the chain rule, intuitively?

If A affects B, and B affects C, then the effect of A on C is the effect of A on B multiplied by the effect of B on C. If raising your speed by 1 km/h saves 2 minutes, and each minute saved earns 10 points, then 1 km/h earns 20 points. A neural network is a long chain of such dependencies: weight → weighted sum → activation → next layer → … → loss. Backpropagation multiplies these local rates of change from the loss backwards to each weight.

What does a tiny worked example look like?

One neuron, no activation: prediction ŷ = w·x, loss L = (ŷ − y)². Let x = 2, y = 10 and current weight w = 3.

  1. Forward: ŷ = 3 × 2 = 6. Loss = (6 − 10)² = 16.
  2. Backward, step 1: how does the loss change with ŷ? dL/dŷ = 2(ŷ − y) = 2 × (−4) = −8.
  3. Backward, step 2: how does ŷ change with w? dŷ/dw = x = 2.
  4. Chain rule: dL/dw = −8 × 2 = −16. The negative sign says increasing w would reduce the loss.
  5. Update: with learning rate 0.05, w ← 3 − 0.05 × (−16) = 3.8.
StepWeight wPredictionLoss
03.006.0016.00
13.807.605.76
24.288.562.07
34.579.140.75
Loss falling over three gradient-descent steps
Step 016.0
Step 15.76
Step 22.07
Step 30.75

Computed from the example; the weight approaches 5, where the prediction equals the target.

A real network does exactly this, but for every weight in every layer at once, using matrices.

How does it work across several layers?

Gradients flow backwards layer by layer. For each layer, backpropagation computes two things: the gradient for that layer's own weights (used for the update) and the gradient with respect to that layer's inputs (passed back to the previous layer as its "error signal"). Each layer only needs its stored forward values and the signal from the layer above, which is why the whole computation is efficient and can run on GPUs as big matrix multiplications.

What is automatic differentiation?

You rarely write backpropagation by hand. Frameworks such as PyTorch and TensorFlow record the operations in the forward pass as a computation graph, and when you call loss.backward() they apply the chain rule through that graph automatically. Understanding what happens underneath still matters: it explains why some operations break gradients, why memory use grows with batch size and depth (stored activations), and how to read the gradient problems below.

What goes wrong during training?

ProblemSymptomCommon fixes
Vanishing gradientsEarly layers barely change; training stallsReLU-family activations, good initialisation, normalisation, residual connections
Exploding gradientsLoss jumps to infinity or NaNGradient clipping, lower learning rate, normalisation
Learning rate too highLoss oscillates or divergesLower it; use warm-up and schedules
Learning rate too lowLoss falls painfully slowlyRaise it; use an adaptive optimiser such as Adam
Dead ReLUsMany neurons always output 0Leaky ReLU, lower learning rate

Our guide to deep learning training techniques covers normalisation, dropout and optimisers in more depth.

How do optimisers use the gradients?

Plain stochastic gradient descent (SGD) subtracts learning rate × gradient, using a small batch of examples per step. Momentum smooths updates by remembering previous directions. Adam keeps running estimates of each gradient's average and variance to adapt the step size per weight; it is the default choice for many deep learning and LLM training runs. Backpropagation provides the gradients for all of them.

Why should you implement it from scratch once?

Writing a two-layer network with forward and backward passes in NumPy, and checking your gradients against a numerical estimate, turns backpropagation from magic into mechanics. You'll understand why shapes must match, why activations are cached, and what a gradient of NaN means. Program Zero's deep learning course starts exactly there, with an ANN, forward pass and backprop built from scratch, before moving to PyTorch, CNNs and RNNs and LSTMs.

How do you check that gradients are correct?

When you implement backpropagation yourself, bugs are easy to make and hard to see: the network still trains, just badly. Gradient checking catches them. For a few weights, nudge the weight up and down by a tiny amount ε, measure the change in loss, and compute the numerical gradient (L(w + ε) − L(w − ε)) ÷ 2ε. Compare it with your backpropagated gradient; they should agree to several decimal places. Frameworks do this for you in their tests, but doing it once by hand builds real confidence that you understand what the backward pass computes.

What does backpropagation cost in memory?

The backward pass needs the activations from the forward pass, so training uses far more memory than inference: every layer's outputs for every example in the batch must be kept until the backward pass reaches them. This is why GPU memory limits batch size and model size during training. Techniques such as gradient checkpointing (recomputing some activations instead of storing them), mixed-precision arithmetic and splitting models across GPUs all exist to manage this memory, and they become essential when training large language models.

Understanding these costs also explains why fine-tuning methods like LoRA save so much memory: frozen weights need no gradients or optimiser state, only the small adapter matrices do.

How does this connect to large language models?

An LLM is a very deep network of transformer layers, trained by predicting the next token with cross-entropy loss. Every training step is a forward pass, a loss, a backward pass computing gradients for billions of weights, and an optimiser update, the same loop as the one-neuron example above. Fine-tuning methods such as LoRA simply restrict which weights receive updates. Master the small case and the large one becomes an engineering question rather than a mystery.