Neural networks train well when a handful of techniques work together: sensible weight initialisation, ReLU-family activations, an adaptive optimiser such as AdamW with a learning-rate schedule, normalisation layers, residual connections for depth, and regularisation such as weight decay, dropout, data augmentation and early stopping. When training fails, a systematic checklist, starting with overfitting a tiny batch, finds the cause faster than guessing.

Why do these techniques matter?

The backpropagation algorithm tells you which direction to move each weight, but not how to make that process fast, stable and generalisable. Deep networks are notorious for training badly without care: gradients vanish or explode, losses stall or spike, and models memorise the training set. The techniques below are the accumulated engineering wisdom that turned deep learning from fragile research into a dependable tool.

How should weights be initialised?

If all weights start equal, every neuron learns the same thing; if they're too large or too small, signals explode or vanish as they pass through layers. Modern initialisations scale random weights according to layer size: Xavier/Glorot for tanh and sigmoid activations, He/Kaiming for ReLU. Frameworks apply sensible defaults, but knowing why they exist helps when you build custom layers.

Which activation functions should you use?

ActivationFormula (idea)Notes
SigmoidSquashes to 0–1Output layer for binary probability; saturates in hidden layers
TanhSquashes to −1–1Used in RNNs; can saturate
ReLUmax(0, x)Simple, fast default; can create "dead" neurons
Leaky ReLUSmall slope for negativesAvoids dead neurons
GELU / SiLU (Swish)Smooth ReLU-like curvesCommon in transformers and modern CNNs
SoftmaxProbabilities over classesOutput layer for multi-class problems

Which optimisers matter?

OptimiserIdeaTypical use
SGDStep against the gradient on mini-batchesBaseline
SGD + momentumAccumulates direction to smooth noisy stepsMany vision models
RMSPropScales steps by recent gradient magnitudeOlder RNN work
AdamMomentum plus per-weight adaptive step sizesStrong general default
AdamWAdam with decoupled weight decayTransformers and LLM training

How should the learning rate change during training?

The learning rate is the single most important hyperparameter. Too high and the loss diverges; too low and training crawls. Schedules help:

  • Warm-up: start small and increase over the first few hundred or thousand steps, which stabilises early training, especially for transformers.
  • Decay: reduce the rate over time (step, exponential, or cosine decay) so the model settles into a good solution.
  • Reduce on plateau: cut the rate when validation loss stops improving.
  • Learning-rate finder: sweep rates in a short run and pick one just below where the loss starts rising sharply.
A warm-up then cosine-decay schedule, as a share of the peak learning rate
Step 0 (start)~0%
End of warm-up100%
Halfway through decay50%
End of training~0%

Cosine decay: rate = peak × (1 + cos(π × progress)) ÷ 2, giving 50% at the halfway point.

What do normalisation layers do?

Batch normalisation normalises each feature across the batch, then rescales with learned parameters. It allows higher learning rates and faster convergence, and is standard in CNNs. It behaves differently in training and inference (using running averages), a common source of bugs. Layer normalisation normalises across features within each example, independent of batch size, which is why transformers and RNNs use it. RMSNorm, a simpler variant, is common in modern LLMs.

Why do residual connections help?

A residual connection adds a block's input to its output: output = x + F(x). Gradients can flow straight through the addition, so very deep networks stay trainable, and each block only needs to learn a small correction. Introduced in ResNet, residual connections are now in essentially every deep architecture, including every transformer layer.

Which regularisation techniques prevent overfitting?

  • Weight decay (L2): discourages large weights; use AdamW's decoupled version with Adam.
  • Dropout: randomly zeroes activations during training (for example 10–50%); turned off at inference.
  • Data augmentation: flips, crops, colour changes for images; noise and masking for audio and text.
  • Early stopping: stop when validation loss stops improving and keep the best checkpoint.
  • Label smoothing: soften one-hot targets slightly to reduce over-confidence.
  • Transfer learning: start from pretrained weights; see transfer learning explained.

How do you make training faster and fit in memory?

  • Mixed precision: compute in 16-bit formats (FP16 or BF16) where safe, roughly halving memory and speeding up modern GPUs.
  • Gradient accumulation: simulate a large batch by summing gradients over several small batches.
  • Gradient checkpointing: recompute some activations during the backward pass instead of storing them.
  • Gradient clipping: cap gradient norms to prevent explosions, standard for RNNs and transformers.
  • Efficient data loading: keep the GPU busy with parallel loaders and pre-processed data.

What should you do when training goes wrong?

  1. Overfit a tiny batch (say 10 examples). If the loss can't reach near zero, there's a bug in the model, loss or data pipeline.
  2. Check data and labels: visualise inputs after preprocessing; verify labels match.
  3. Check the loss function matches the output layer (for example, raw logits with cross-entropy, not softmax twice).
  4. Tune the learning rate before anything else.
  5. Watch gradient norms for vanishing or exploding values.
  6. Compare training and validation curves to diagnose under- versus overfitting.
  7. Change one thing at a time and log every experiment (see MLflow experiment tracking).

How do you read training curves?

Plot training and validation loss after every epoch. If both fall together and flatten, training is healthy; consider a bigger model or longer training for more accuracy. If training loss keeps falling while validation loss rises, the model is overfitting: add regularisation, augmentation or more data, or stop earlier. If both stay high, the model is underfitting: increase capacity, train longer or revisit the features and learning rate. Sudden spikes usually mean the learning rate is too high or a bad batch slipped in. Learning to read these curves quickly is one of the most valuable practical skills in deep learning.

Why does batch size matter?

Batch size affects speed, memory and results. Larger batches use the GPU more efficiently and give smoother gradient estimates, but need more memory and often a higher learning rate; a common heuristic is to scale the learning rate roughly with the batch size, with warm-up. Very large batches can generalise slightly worse without adjustments. Smaller batches add noise to each update, which sometimes helps find better solutions but slows training. In practice, choose the largest batch that fits in memory (using gradient accumulation if you need more), then tune the learning rate for that batch size rather than reusing a value tuned for another.

Should you train with more data or a bigger model?

When a model underfits, a bigger model often helps; when it overfits, more or more varied data usually helps more than any regularisation trick. Research on scaling laws (see scaling laws explained) shows that model size, data and compute must grow together for the best results.

How should you practise these techniques?

Train the same small CNN several times: plain SGD versus AdamW, with and without batch normalisation, with and without dropout and augmentation, with and without a schedule. Plot every run's validation curve on one chart. These experiments teach more than any table. Program Zero's deep learning course covers these techniques in PyTorch and TensorFlow/Keras, and they return at scale in the LLM course, where you train a GPT-style model from scratch.