A GPT is a decoder-only transformer trained to predict the next token. To build a small one in PyTorch: prepare a text corpus, build a tokenizer, embed tokens and positions, stack transformer blocks (causal multi-head self-attention plus a feed-forward network, each with residual connections and layer normalisation), add an output layer over the vocabulary, train with cross-entropy on next-token prediction, and generate text by sampling one token at a time. A model of a few million parameters is enough to learn how the large ones work.

Why build a GPT yourself?

Using an LLM API teaches you prompting. Building a small GPT teaches you what the model actually is: why context length is limited, what temperature does, why tokenization matters, why models hallucinate, and what "parameters" and "training" mean in practice. Engineers who have built one debug real LLM systems faster and make better decisions about fine-tuning, retrieval and cost. It's also a satisfying project that shows up well in a portfolio, because it demonstrates understanding rather than API usage.

What are the components, in order?

StepComponentWhat it does
1DatasetA text corpus split into training and validation parts
2TokenizerTurns text into integer token IDs and back
3Token + position embeddingsTurns IDs into vectors that also encode position
4Causal self-attentionLets each token gather context from earlier tokens only
5Feed-forward networkProcesses each position's vector independently
6Transformer blockAttention + feed-forward with residuals and layer norm
7Output headProjects to logits over the vocabulary
8Training loopCross-entropy on next-token prediction, AdamW
9GenerationSample tokens one at a time with temperature

Step 1–2: how do you prepare data and a tokenizer?

Pick a text corpus you're allowed to use: public-domain books, your own writing, or an open dataset. A few megabytes is plenty for a first model. The simplest tokenizer is character-level: every distinct character gets an ID. It's trivial to build and makes the model's job harder but the code easier. Real LLMs use subword tokenizers such as byte-pair encoding (BPE), which merge frequent character sequences into single tokens so common words take one token and rare words are split into pieces. Start with characters, then swap in a BPE tokenizer as an upgrade. Split the data 90/10 into training and validation sets.

Step 3: what are embeddings doing?

A token embedding table maps each token ID to a learned vector of size d (for example, 128 or 256). Because attention ignores order on its own, you also add a position embedding: either a learned vector per position (as in the original GPT models) or a rotary scheme like RoPE used in many modern LLMs. The sum of token and position embeddings is the input to the first transformer block. The block size (context length) is the maximum number of tokens the model sees at once.

Step 4: how does causal self-attention work?

Each token's vector is projected into a query, key and value. Attention scores are query-key dot products scaled by √d; a causal mask sets scores for future positions to negative infinity so each token can only look backwards; softmax turns scores into weights; and the output is the weighted sum of values. Multi-head attention runs several smaller attention heads in parallel and concatenates them. If any of this is unfamiliar, read the attention mechanism explained first; it is the heart of the model.

Step 5–6: what goes into a transformer block?

A block applies layer normalisation, then attention, then adds the result back to the input (a residual connection); then layer normalisation, a two-layer feed-forward network (typically expanding to 4 × d with a GELU activation and projecting back), and another residual connection. This "pre-norm" arrangement trains more stably than the original post-norm design. Stack several identical blocks: 4 to 6 for a small model. Dropout inside attention and feed-forward layers helps on small datasets.

Step 7–8: how is the model trained?

The output layer maps each position's final vector to logits over the vocabulary. Training data is created by taking random chunks of block-size tokens as inputs and the same chunks shifted by one token as targets, so every position learns to predict the next token. The loss is cross-entropy between logits and targets. Use AdamW, a learning rate around 3e-4 to 1e-3 for small models with warm-up, gradient clipping, and evaluate the validation loss regularly.

Parameter counts for small GPT configurations (approximate, vocabulary of 65 characters)
4 layers, d = 128≈0.8M
6 layers, d = 256≈4.8M
6 layers, d = 384≈10.7M

Arithmetic: each block has about 12 × d² weights (4d² in attention, 8d² in the feed-forward layer); embeddings add a small amount for a character vocabulary.

A useful sanity check: before training, the loss should be close to ln(vocabulary size), about 4.2 for 65 characters, because the untrained model guesses uniformly. If it starts far higher, something is wrong with initialisation or the loss.

Step 9: how does generation work?

Start with a prompt (or a single newline token). Feed it through the model, take the logits for the last position, divide by a temperature (below 1 for safer text, above 1 for more variety), optionally keep only the top-k tokens, apply softmax, sample one token, append it, and repeat. Crop the input to the block size as it grows. Watching samples improve from noise to plausible words to readable text in the style of your corpus is the moment it all clicks.

What results should you expect?

Training stageTypical samples (character-level)
UntrainedRandom characters
EarlyWord-like clusters, spaces and punctuation in roughly the right places
MiddleReal common words, occasional short phrases
LaterSentence structure and style of the corpus; content often nonsensical

That last row is instructive: fluent but meaningless text is exactly how hallucination starts, just at a much smaller scale.

What should you add next?

  • A BPE tokenizer and a larger corpus.
  • Mixed precision and a GPU for larger models.
  • Modern components: RoPE, RMSNorm, SwiGLU feed-forward layers and a key-value cache for fast generation; see modern LLM architecture.
  • Fine-tuning: train on instruction-response pairs to make it follow simple instructions.
  • Evaluation: track validation loss and perplexity.

What mistakes do people make?

  • Forgetting the causal mask, so the model "cheats" by seeing the answer and training loss collapses suspiciously fast.
  • Off-by-one errors when shifting targets.
  • Training only on the training set and never checking validation loss, missing overfitting.
  • A learning rate far too high for the model size, causing NaN losses.
  • Mixing up tensor shapes between (batch, time, channels) and (batch, heads, time, head size).

How do you know your model is learning well?

Watch the training and validation losses together. Both should fall; when validation loss flattens or starts rising while training loss keeps falling, the model is memorising your small corpus, so stop there or add dropout and data. Perplexity, the exponential of the loss, gives an intuitive reading: a perplexity of 5 means the model is, on average, about as uncertain as choosing among five equally likely characters.

Where does this sit in a learning path?

You need Python, NumPy, PyTorch and an understanding of backpropagation and attention first. In Program Zero, building a GPT-style LLM from scratch in PyTorch is a core part of the generative AI and LLM course, and the phase project is a small GPT trained from scratch plus a LoRA fine-tune of an open model. Read how LLMs work for how this connects to production models.