BERT is a transformer encoder that learns to understand text by reading it in both directions at once. It was pretrained mainly by masked language modelling (predicting hidden words from the words on both sides), and it is fine-tuned for tasks such as classification, named-entity recognition, question answering and search ranking. Unlike GPT-style models, it isn't built to generate long text, but for understanding tasks it is accurate, compact and fast, which keeps it widely used in production.

Why was BERT a breakthrough?

Before BERT, most NLP systems were trained from scratch for each task or used word embeddings that gave every word one fixed meaning regardless of context ("bank" was the same vector in "river bank" and "bank loan"). Released by Google in 2018, BERT showed that pretraining one deep, bidirectional transformer on large amounts of unlabelled text, then fine-tuning it briefly for each task, beat specialised models across a wide range of benchmarks. It made transfer learning the standard approach in NLP and led directly to today's foundation models.

What does "bidirectional" mean?

A GPT-style decoder reads left to right: when processing a word, it sees only the words before it, because it must predict what comes next. BERT's encoder uses self-attention without a causal mask, so every word attends to every other word in the input, before and after it. To understand "bank" in "She sat on the bank and watched the river", seeing "river" later in the sentence helps. For understanding tasks, where the whole text is available, this two-way context is a major advantage. See the attention mechanism explained for the masking details.

How was BERT pretrained?

Masked language modelling (MLM): 15% of input tokens are selected; most are replaced with a [MASK] token, some with a random token and some left unchanged, and the model learns to predict the originals using context on both sides. Next sentence prediction (NSP): given two sentences, predict whether the second actually followed the first. Later research (notably RoBERTa) found NSP contributed little and that more data and longer training mattered more.

What sizes does BERT come in?

ModelLayersHidden sizeParameters (approx.)
BERT-base12768110 million
BERT-large241,024340 million
DistilBERT676866 million
Parameters in common encoder models (millions)
DistilBERT66
BERT-base110
BERT-large340

Published model sizes. For comparison, many current generative LLMs have billions of parameters.

How is BERT fine-tuned for tasks?

TaskHow BERT is usedExample
Text classificationA small classifier on the [CLS] token's final vectorSentiment, spam, ticket routing
Named-entity recognitionA label for every tokenFinding names, places, amounts in documents
Extractive question answeringPredict start and end positions of the answer spanAnswering from a policy document
Sentence-pair tasksBoth sentences in one input, separated by [SEP]Duplicate-question detection, entailment
Embeddings and searchEncoders fine-tuned to produce sentence vectorsSemantic search, RAG retrieval, clustering

Fine-tuning typically takes a few epochs with a small learning rate (around 2e-5 to 5e-5). With the Hugging Face Transformers library, loading a pretrained BERT and fine-tuning it for classification takes a few dozen lines of code.

What are the special tokens for?

[CLS] is placed at the start of every input; its final vector serves as a summary of the whole sequence for classification. [SEP] separates two segments, such as a question and a passage. [MASK] marks hidden tokens during pretraining. [PAD] fills shorter sequences to a common length in a batch, and an attention mask tells the model to ignore padding. BERT uses a WordPiece tokenizer, which splits rare words into subword pieces.

Which BERT variants should you know?

  • RoBERTa: more data, longer training, no NSP; generally stronger.
  • DistilBERT: a smaller, faster model trained by distillation, keeping most of BERT's accuracy.
  • ALBERT: parameter sharing to reduce size.
  • DeBERTa: improved attention that separates content and position; strong on benchmarks.
  • Multilingual models: mBERT, XLM-RoBERTa and Indian-language models such as IndicBERT and MuRIL, trained on Indian languages.
  • Sentence encoders: models fine-tuned to produce good sentence embeddings for search and RAG.

How does BERT compare with GPT-style models?

AspectBERT (encoder)GPT (decoder)
Reads textBoth directionsLeft to right
Pretraining taskMasked word predictionNext-token prediction
Best atClassification, extraction, searchGeneration, conversation, reasoning
Typical size in productionTens to hundreds of millions of parametersBillions of parameters
Cost per predictionLowHigher

A practical rule: if you need a label, a span or an embedding at high volume and low cost, a fine-tuned encoder is often the better engineering choice. If you need open-ended generation or flexible reasoning, use a generative model. Many systems use both, for example an encoder for retrieval and a generative model for the answer.

What are BERT's limitations?

  • Limited input length (512 tokens in the original model), so long documents must be split or handled with long-context variants.
  • Not a text generator in the usual sense.
  • Needs labelled data for each task, unlike instruction-following LLMs that can attempt tasks from a prompt.
  • Inherits biases from its pretraining text.

How does BERT produce embeddings for search and RAG?

A raw BERT model's outputs aren't ideal sentence embeddings out of the box: averaging token vectors or using [CLS] gives only moderate results for similarity. Sentence-embedding models fix this by fine-tuning BERT-style encoders on pairs of related sentences (questions and answers, paraphrases) so that similar meanings end up close together. These encoders power semantic search and the retrieval step in RAG systems, often running far more cheaply than a generative model. A common production architecture uses a small encoder to find relevant passages and a generative LLM only to write the final answer.

What does fine-tuning BERT look like in practice?

Suppose you want to route customer emails to five teams. You label around 2,000 past emails, split them into training, validation and test sets, tokenise them with the model's tokenizer (truncating to a sensible length), and fine-tune a model such as DistilBERT for three epochs with a learning rate around 3e-5. You evaluate macro-F1 on the validation set, inspect the confusion matrix to see which teams get mixed up, fix any mislabelled examples you find, and retrain. On a single GPU this takes minutes, and the resulting model can classify many emails per second on modest hardware, far more cheaply than sending each email to a large generative model.

How is BERT asked about in interviews?

Expect questions on why bidirectional context helps understanding, how masked language modelling works and why some masked tokens are replaced with random words or left unchanged, what the [CLS] token is for, how you'd fine-tune for classification versus token tagging, and when you'd choose an encoder over a generative LLM. Strong answers mention cost and latency, not just accuracy, and can describe a project where they actually fine-tuned an encoder and measured the result against a simpler baseline such as TF-IDF with logistic regression.

How should you practise?

Fine-tune DistilBERT on a sentiment or ticket-classification dataset, then a token-classification model for named entities, and compare with a TF-IDF plus logistic regression baseline. Encoder models and fine-tuning with Hugging Face are part of Program Zero's NLP course, alongside building a GPT from scratch; see building a GPT in PyTorch for the decoder side.