Modern large language models keep the core decoder-only transformer from 2017 but change many details: pre-normalisation with RMSNorm for stable training, SwiGLU feed-forward layers, rotary position embeddings (RoPE) for better handling of position and length, grouped-query attention (GQA) and key-value caching for cheaper generation, memory-efficient attention kernels such as FlashAttention, and sometimes mixture-of-experts (MoE) layers to grow capacity without growing compute per token.

What stayed the same since the original transformer?

The basic recipe is unchanged: token embeddings, a stack of blocks each combining self-attention and a feed-forward network with residual connections, and an output layer predicting the next token, trained with cross-entropy. If you've built a small GPT (see building a GPT from scratch), you already understand the skeleton. What changed are the components inside, each driven by a concrete problem: training stability at scale, quality at long context lengths, and above all the cost of serving models to many users.

What are the main upgrades?

ComponentOriginal transformerCommon in modern LLMsWhy
Normalisation placementPost-normPre-normMore stable training of deep stacks
Normalisation typeLayerNormRMSNormSimpler, slightly faster, works as well
Feed-forward activationReLU / GELUSwiGLUBetter quality per parameter
Position encodingFixed sinusoidal or learnedRotary (RoPE)Relative positions; easier context extension
Attention headsMulti-head (separate K, V per head)Grouped-query attentionSmaller KV cache, faster generation
Attention kernelStandard implementationFlashAttention-style kernelsLess memory traffic, longer contexts
Feed-forward layerDenseDense or mixture-of-expertsMore capacity at similar compute

Why pre-norm and RMSNorm?

In the original design, normalisation came after adding the residual (post-norm). Placing it before each sub-layer (pre-norm) keeps the residual path clean, so gradients flow more easily through very deep stacks and training needs less delicate warm-up. RMSNorm simplifies LayerNorm by scaling activations by their root mean square, skipping the mean subtraction and bias. It's cheaper and in practice works as well, so many recent models use it.

What is SwiGLU?

The feed-forward network in each block expands each token's vector, applies a non-linearity and projects it back. SwiGLU uses a gated linear unit with the Swish (SiLU) activation: one projection is passed through the activation and multiplied element-wise by a second projection, which acts as a learned gate. Experiments showed better quality for the same compute, so it became standard in many model families. Because it uses three weight matrices instead of two, the hidden size is usually reduced to keep the parameter count similar.

How do rotary position embeddings work?

RoPE encodes position by rotating each query and key vector by an angle that depends on its position, with different frequencies across dimensions. When a query and key are compared with a dot product, the result depends on the difference between their positions, so attention naturally captures relative distance. RoPE adds no extra parameters, and techniques that adjust its frequencies help extend a model to longer contexts than it was trained on, an important practical advantage.

Why does the KV cache matter so much?

Generation is sequential: each new token attends to all previous ones. Without caching, every step would recompute keys and values for the whole sequence. The KV cache stores them, so each step only computes the new token's. But the cache grows with context length, number of layers and number of heads, and at long contexts and large batch sizes it can use more GPU memory than the model weights. That memory cost is the main driver behind grouped-query attention.

Relative KV-cache size for a layer with 32 query heads
Multi-head (32 KV heads)1×
GQA with 8 KV heads0.25×
Multi-query (1 KV head)0.03×

Arithmetic: cache size is proportional to the number of key-value heads (8 ÷ 32 = 0.25; 1 ÷ 32 ≈ 0.03).

What are multi-query and grouped-query attention?

In standard multi-head attention, every head has its own keys and values. Multi-query attention shares one set of keys and values across all heads, shrinking the cache dramatically but sometimes hurting quality. Grouped-query attention (2023) is the middle ground: query heads are split into groups that share keys and values, for example 32 query heads sharing 8 KV heads. It keeps quality close to multi-head attention while cutting cache memory and speeding generation, so it's widely adopted.

What does FlashAttention change?

Standard attention writes the full score matrix (sequence length × sequence length) to GPU memory, which is slow and memory-hungry because moving data, not arithmetic, is the bottleneck. FlashAttention (2022) computes exact attention in tiles that stay in fast on-chip memory, never storing the full matrix. The result is the same, but faster and with memory that grows linearly rather than quadratically in sequence length for the stored activations. It's an example of how hardware-aware engineering matters as much as model design. See the attention mechanism explained for the underlying maths.

How do mixture-of-experts models work?

An MoE layer replaces one feed-forward network with many expert networks plus a small router that sends each token to the top few experts (often two). Total parameters can be very large while each token uses only a fraction, so compute per token stays modest. The challenges are balancing load across experts, communication between GPUs that hold different experts, and memory: all experts must still be stored even though few run per token.

What else matters at inference time?

  • Quantization: storing weights in 8-bit or 4-bit formats to cut memory and cost.
  • Continuous batching and paged KV memory: serving engines such as vLLM pack many requests efficiently on one GPU.
  • Speculative decoding: a small draft model proposes tokens that the large model verifies in parallel.
  • Sliding-window attention: limiting attention to recent tokens in some layers for long inputs.

How do these choices affect cost per answer?

For a company serving a chatbot, the architecture decides the bill. A model with grouped-query attention fits more simultaneous conversations on one GPU because each conversation's cache is smaller. Quantised weights free memory for more cache. A mixture-of-experts model can answer with quality closer to a much larger dense model while computing far less per token, if the deployment can hold all experts in memory. Engineers who understand these trade-offs can cut serving costs substantially by choosing the right model and serving setup, without touching the product itself.

How do you compare architectures fairly?

Change one component at a time, keep parameter count and training tokens equal, use the same data and tokenizer, and compare validation loss plus a few downstream tasks. Many published improvements are small, so run several seeds before trusting a difference.

Which of these should you learn first?

Start with the KV cache, because it explains most of the cost of running LLMs and is simple to implement. Then RoPE, which every engineer extending context length will meet. Then grouped-query attention and quantization, which decide how many users one GPU can serve. Mixture-of-experts and custom attention kernels can come later, once the basics feel natural.

How do you learn this hands-on?

Start from a small GPT you've built, then swap in RMSNorm, SwiGLU and RoPE one at a time and compare validation loss; add a KV cache and measure generation speed; implement GQA and compare cache size. Modern architectures (RoPE, GQA, MoE) and inference optimisation (quantization, vLLM) are part of Program Zero's advanced LLM engineering phase in the generative AI and LLM course. Read scaling laws explained for the bigger picture of why these efficiencies matter.