Attention lets a neural network decide, for each word in a sequence, which other words matter most for understanding it. Each token is projected into a query, a key and a value; the similarity between a token's query and every key becomes a set of weights (via softmax), and the token's new representation is the weighted sum of the values. Self-attention, stacked in many layers and heads, is the core of the transformer and of every modern large language model.
What problem did attention solve?
Before transformers, sequence models such as RNNs and LSTMs read text one token at a time and squeezed everything seen so far into a single hidden state. Long sentences overwhelmed that bottleneck: in translation, details from the start of a long sentence faded by the end. Attention, introduced for translation in 2014 by Bahdanau and colleagues, let the decoder look back at all encoder states and focus on the relevant ones for each output word. The 2017 paper "Attention Is All You Need" then showed that attention alone, without recurrence, could build better and far more parallel models: the transformer.
What is the intuition behind attention?
Read: "The bank raised its interest rate because it was worried about inflation." To understand it, you look back and connect it with bank. To understand bank (financial, not river) you lean on interest rate. Attention gives every word a learned way to look at every other word and pull in the context it needs. After a few layers of this, each word's vector carries meaning from its whole sentence.
What are queries, keys and values?
Each token's embedding is multiplied by three learned matrices to create:
| Vector | Role | Analogy |
|---|---|---|
| Query (Q) | What this token is looking for | A search query you type |
| Key (K) | What each token offers to be matched on | Titles of documents in an index |
| Value (V) | The information each token passes on | The documents' contents |
A token compares its query with every key (dot products), turns the scores into weights that sum to 1 with softmax, and takes the weighted average of all values. Because Q, K and V come from learned projections, the model learns what to look for and what to offer during training.
What is scaled dot-product attention?
In matrix form: Attention(Q, K, V) = softmax(QKᵀ ÷ √d) · V, where d is the size of the key vectors. Dividing by √d keeps the scores from growing too large as vectors get longer; without it, softmax would become extremely peaked and gradients would vanish. Everything here is matrix multiplication, which GPUs do extremely fast, and all tokens are processed in parallel, unlike an RNN's step-by-step loop.
What does a tiny example look like?
Suppose, after scaling, the token "it" has scores of 2.0 for "bank", 0.5 for "rate", 0.0 for "worried" and −1.0 for "the". Softmax converts scores to weights:
Computed: e² ≈ 7.39, e^0.5 ≈ 1.65, e⁰ = 1, e⁻¹ ≈ 0.37; each divided by their sum, ≈ 10.41.
The new vector for "it" is about 70% bank's value vector, 16% rate's and so on: its representation now carries the information that it refers to the bank.
What is multi-head attention?
One set of Q, K and V captures one kind of relationship. Multi-head attention runs several attention operations in parallel, each with its own projections, then concatenates and mixes the results. Different heads learn different patterns: one might link pronouns to nouns, another track syntax, another connect a verb with its object. Large models use many heads per layer and dozens of layers.
What is masking, and why do LLMs need it?
A language model is trained to predict the next token, so it must not peek at future tokens. Causal masking sets the scores for future positions to negative infinity before softmax, giving them zero weight. That's the difference between a decoder-only model like GPT, which uses causal self-attention, and an encoder like BERT, which attends in both directions. See BERT explained and building a GPT from scratch for both sides.
How do the attention types compare?
| Type | Queries from | Keys/values from | Used in |
|---|---|---|---|
| Bidirectional self-attention | Same sequence | Same sequence, all positions | BERT-style encoders |
| Causal self-attention | Same sequence | Same sequence, earlier positions only | GPT-style LLMs |
| Cross-attention | Decoder | Encoder output | Translation, speech-to-text, image captioning |
Why is attention O(n²), and how is that handled?
Each of n tokens compares with all n tokens, so attention needs about n² scores per head per layer. Going from a 4,000-token to a 32,000-token context multiplies that work by 64. Engineers attack this cost in several ways: memory-efficient exact kernels such as FlashAttention, which avoid storing the full score matrix; key-value caching during generation, so past tokens aren't recomputed; grouped-query attention, which shares keys and values across heads to save memory; and sparse or sliding-window attention for very long inputs. Our post on modern LLM architecture covers these in detail, and Big O notation explained covers the complexity basics.
How does position information get in?
Attention by itself treats its input as a set: shuffle the words and each word gets the same weighted mix. Transformers therefore add positional information, originally fixed sine and cosine patterns added to embeddings, and in most modern LLMs rotary position embeddings (RoPE), which rotate queries and keys by an angle depending on position so attention scores reflect relative distance.
Can you see what attention heads are doing?
Attention weights are easy to visualise as a grid: rows are the tokens doing the looking, columns the tokens being looked at, and darker cells mean higher weight. In trained models, some heads clearly track the previous token, some link pronouns with the nouns they refer to, and some connect matching brackets in code. Be careful not to over-read these pictures, though. Attention weights show where information flows in one layer, not a complete explanation of the model's decision, and researchers use more rigorous interpretability methods when they need real answers.
Where else is attention used?
Attention is not only for text. Vision transformers apply it to image patches, speech models to audio frames, and multimodal models use cross-attention to connect text with images. Recommendation systems use attention over a user's past actions to weigh which ones predict the next. Wherever a model must decide which parts of a large input matter for each output, attention is now the default tool, which is why it is worth understanding at the level of the matrix maths rather than only the analogy.
How should you learn attention properly?
- Compute attention for a four-word sentence by hand with tiny 2-dimensional vectors.
- Implement scaled dot-product attention in NumPy, then multi-head attention in PyTorch.
- Add a causal mask and check that outputs don't change when you alter future tokens.
- Build a small transformer block (attention, feed-forward, residuals, layer norm) and train it on a toy task.
Program Zero derives attention and builds the transformer from scratch in its generative AI and LLM course, then trains a small GPT-style model end to end. The NLP course page covers the language foundations that come first.