A recurrent neural network (RNN) reads a sequence one element at a time and carries a hidden state, a running summary of what it has seen, from step to step. Plain RNNs forget long-range information because gradients vanish over many steps; LSTMs and GRUs add learned gates that decide what to keep, forget and output, so they can remember over much longer spans. Transformers have replaced them for most language tasks, but they remain useful for time series and streaming data, and they explain why attention was such a breakthrough.
Why do sequences need special networks?
In a sentence, a stock-price series or a stream of sensor readings, order matters and inputs vary in length. "The film was not good" and "The film was good, not bad" use similar words with opposite meanings. A plain feed-forward network takes a fixed-size input with no notion of order, so it can't naturally process "the next word given everything before". Sequence models need to carry context forward through time.
How does an RNN work?
At each time step t, the RNN takes the current input xₜ and the previous hidden state hₜ₋₁ and computes a new hidden state: hₜ = tanh(W·xₜ + U·hₜ₋₁ + b). The same weights W and U are reused at every step, so the network can handle sequences of any length. The hidden state can produce an output at every step (tagging each word) or only at the end (classifying the whole sentence). Picture the network "unrolled" in time: a chain of identical cells, each passing its state to the next.
How are RNNs trained?
With backpropagation through time: unroll the network over the sequence, compute the loss, and apply the chain rule backwards through every time step. Because the same weights appear at every step, their gradients are summed across steps. For long sequences, the unrolled network is very deep, which brings the classic deep-network problems of vanishing and exploding gradients.
Why do plain RNNs forget?
Gradients flowing back through many steps are multiplied repeatedly by factors from the recurrent weights and activation derivatives. If those factors are mostly below 1, the gradient shrinks exponentially; above 1, it explodes.
Arithmetic: 0.9^10 ≈ 0.35, 0.9^25 ≈ 0.072, 0.9^50 ≈ 0.0052.
So a plain RNN learns well from the last few words but barely from words 50 steps back, which is exactly where some important context (the subject of a long sentence) may be. Gradient clipping handles exploding gradients; vanishing gradients needed a new architecture.
How does an LSTM fix this?
The Long Short-Term Memory network (Hochreiter and Schmidhuber, 1997) adds a cell state, a separate memory line running through time that is changed only by small, controlled updates, plus three learned gates (each a sigmoid output between 0 and 1):
| Gate | Decides | Example |
|---|---|---|
| Forget gate | What to erase from the cell state | Forget the old subject when a new sentence starts |
| Input gate | What new information to write | Store that the new subject is plural |
| Output gate | What part of the memory to expose as the hidden state | Reveal "plural" when choosing the verb form |
Because the cell state is updated mostly by addition rather than repeated multiplication, gradients can flow back across many steps without vanishing, the same insight behind residual connections in CNNs and transformers.
What is a GRU?
The Gated Recurrent Unit (2014) simplifies the LSTM: it merges the forget and input gates into an update gate, adds a reset gate, and drops the separate cell state. It has fewer parameters, trains faster and often matches LSTM performance. When in doubt, try both.
What architectures are built from RNNs?
- Many-to-one: sentiment classification of a review.
- Many-to-many (aligned): tagging each word with its part of speech or entity type.
- Sequence-to-sequence (encoder-decoder): an encoder RNN reads a sentence, a decoder RNN writes its translation. This is where attention was first introduced, to let the decoder look back at all encoder states.
- Bidirectional RNNs: read the sequence forwards and backwards, useful when the whole input is available, such as tagging.
- Stacked RNNs: several recurrent layers for richer representations.
How do RNNs compare with transformers?
| Aspect | RNN / LSTM | Transformer |
|---|---|---|
| Processing | Sequential, step by step | All positions in parallel |
| Training speed on GPUs | Slower (can't parallelise over time) | Much faster |
| Long-range context | Limited, even with gates | Direct access to every position |
| Cost per step at inference | Constant memory | Grows with context length |
| Typical use today | Some time series, streaming, small devices | Language, vision, multimodal, most new work |
The constant-memory property is why recurrent ideas keep returning in research, in architectures that try to combine transformer-quality modelling with RNN-like efficiency for very long sequences.
Where are LSTMs still used?
- Time-series forecasting of demand, load or sensor data, often compared with gradient boosting and simpler statistical models.
- Streaming and on-device tasks where inputs arrive continuously and memory is tight.
- Teaching: building a character-level text generator with an LSTM is one of the best ways to understand sequence modelling before transformers.
What does a character-level text generator teach you?
Train an LSTM to predict the next character in a body of text, such as public-domain stories, and then sample from it one character at a time. Early in training it produces random letters; then plausible word shapes; then real words and punctuation; finally passable sentences in the style of the source. Watching this progression teaches the core ideas of language modelling, including next-token prediction, temperature and sampling, the same ideas that drive large language models at vastly greater scale.
What are common mistakes with RNNs?
- Not padding and masking correctly when batching sequences of different lengths.
- Leaking future values in time-series features or validation.
- Skipping gradient clipping and getting NaN losses.
- Using random train/test splits on time series instead of time-ordered splits.
- Feeding raw, unscaled values to the network, which slows or destabilises training.
How are LSTMs used for time-series forecasting?
For forecasting, the sequence is a history of values, such as daily electricity demand, and the target is the next value or the next several. You build training examples with a sliding window (the past 30 days predict the next 7), scale the values, and keep the split strictly time-ordered. LSTMs can capture seasonality and complex patterns, but they don't always win: gradient-boosted models on well-designed lag features, and classical methods such as exponential smoothing, are strong baselines and often easier to maintain. Always compare against a naive forecast (tomorrow equals today, or the same day last week); surprisingly many sophisticated models fail to beat it.
How are RNNs asked about in interviews?
Expect to explain how the hidden state carries information, why vanishing gradients happen in long sequences, what each LSTM gate does, how a GRU differs, and why transformers replaced RNNs for language. Good answers connect the dots: the LSTM's additive cell state and the transformer's residual connections solve the same gradient-flow problem in different ways.
How should you learn sequence models?
Implement a tiny RNN cell in NumPy to see the hidden state update, then use PyTorch's nn.LSTM and nn.GRU for a sentiment classifier and a character-level text generator. Then read about attention and build a small transformer. RNN, LSTM and GRU are part of Program Zero's deep learning course, whose phase project includes an LSTM text generator, leading straight into transformers in the NLP course.