Scaling laws are empirical findings that a language model's loss improves smoothly and predictably as you increase three things together: the number of parameters, the amount of training data, and the compute spent training. Early work (Kaplan and colleagues, 2020) emphasised model size; the Chinchilla study (Hoffmann and colleagues, 2022) showed that for a fixed compute budget, data and parameters should grow roughly in proportion, at about 20 training tokens per parameter. These laws guide how labs plan training runs, and why smaller, longer-trained models are popular for deployment.

What do scaling laws actually describe?

Researchers trained many models of different sizes on different amounts of data and measured their test loss (how well they predict held-out text). Plotted on log-log axes, loss falls along nearly straight lines as each resource grows: a power law. That means each tenfold increase in a resource buys a roughly constant fractional reduction in loss, with diminishing returns in absolute terms. The relationships held across several orders of magnitude, which made them useful for prediction: you could run small experiments and estimate how a much larger model would perform before spending a large budget.

What are the three ingredients?

IngredientSymbolMeaning
Model sizeNNumber of parameters
DataDNumber of training tokens
ComputeCTraining FLOPs, roughly 6 × N × D for transformers

The "6 × N × D" rule of thumb comes from about two FLOPs per parameter per token in the forward pass and about four in the backward pass. It lets you estimate training compute from model size and data size quickly.

What changed between Kaplan and Chinchilla?

The 2020 OpenAI study suggested that, as compute grows, most of it should go into larger models, with data growing more slowly. Following that advice, several very large models were trained on relatively little data. In 2022, DeepMind's Chinchilla paper re-ran the analysis with different methods and found that model size and data should scale about equally. Their 70-billion-parameter Chinchilla model, trained on about 1.4 trillion tokens, outperformed the 280-billion-parameter Gopher, which used similar compute but far less data. The practical message: many large models had been undertrained.

Compute-optimal training tokens at about 20 tokens per parameter (Chinchilla rule of thumb)
1B parameters20B tokens
7B parameters140B
13B parameters260B
70B parameters1.4T

Arithmetic: tokens ≈ 20 × parameters. A rule of thumb from one study, not a law of nature.

Why do many deployed models train far beyond "compute-optimal"?

Chinchilla optimises the cost of training only. But a popular model may answer millions of requests a day, and every request costs inference compute proportional to model size. Training a smaller model on many more tokens than 20 per parameter costs more during training but produces a cheaper, faster model to serve, and loss keeps improving with more data. That's why many widely used open models of a few billion parameters have been trained on trillions of tokens, far beyond the Chinchilla ratio.

What are the limits of scaling?

  • Data supply: high-quality public text is finite; data quality, deduplication and filtering increasingly matter more than raw quantity.
  • Cost: each further improvement needs exponentially more compute.
  • Loss isn't everything: lower loss usually means better ability, but specific skills can appear unevenly, and alignment, instruction tuning and evaluation still decide usefulness.
  • Energy and hardware: large runs need huge GPU clusters and power, a strategic issue that initiatives such as the IndiaAI Mission's compute capacity aim to address in India.

What about emergent abilities?

Some benchmarks show abilities that seem to appear suddenly above a certain scale. Later research suggested that part of this "emergence" depends on how tasks are scored: with all-or-nothing metrics, gradual improvements can look like sudden jumps. Underlying loss improves smoothly, but whether that is useful for a particular task can change quickly near a threshold. The practical lesson is to evaluate on the tasks you care about, at the sizes you can afford.

Do scaling laws apply to fine-tuning and inference?

Similar regularities show up elsewhere: fine-tuning benefits from more and better examples with diminishing returns, larger models often need fewer examples to adapt, and recent work studies "inference-time scaling", where spending more compute while answering (for example, generating and checking several candidate solutions, or reasoning longer) improves results on hard problems. The common thread is that performance often trades off predictably against resources, which turns AI planning into an engineering and budgeting problem.

What does this mean for engineers and companies in India?

SituationImplication
Building an app on existing modelsChoose the smallest model that meets quality targets; serving cost dominates
Domain or Indian-language modelData quality and coverage matter more than adding parameters
Planning a training runRun small-scale experiments and fit curves before committing budget
Limited computeFine-tune strong open models instead of pretraining from scratch

How do researchers fit scaling curves?

The method is straightforward in principle. Train a series of small models across a range of sizes and token counts, record their final validation losses, and fit a function in which loss falls as a power of parameters and data towards an irreducible floor. Then use the fitted curve to choose the best size and data split for a larger budget, and to sanity-check the large run: if its loss lands well above the predicted curve, something is wrong with data, code or hyperparameters. Small, cheap experiments thus protect very expensive ones, which is the most practical lesson scaling laws offer.

What does a worked compute estimate look like?

Suppose you want to train a 1-billion-parameter model on 20 billion tokens, the Chinchilla-style ratio. Training compute is about 6 × 10⁹ × 2 × 10¹⁰ = 1.2 × 10²⁰ floating-point operations. If a GPU sustains, say, 10¹⁴ useful operations per second in practice, that's 1.2 million GPU-seconds, about 333 GPU-hours, or roughly 14 days on one GPU and well under a day on a few dozen. The numbers here are illustrative, but the method is how teams sanity-check plans: estimate FLOPs from parameters and tokens, divide by realistic throughput, then compare with budget and deadline before starting.

The same arithmetic shows why data quality matters so much at a fixed budget: every token you train on costs compute, so spending it on duplicated, low-quality or irrelevant text wastes money as surely as an oversized model does.

How are scaling laws asked about in interviews?

Research-engineer interviews may ask you to estimate training FLOPs with the 6ND rule, explain what Chinchilla changed, and argue whether a given model is under- or over-trained. Applied roles ask the practical version: given a latency and cost budget, which model size would you choose, and how would you test that it's good enough? Answering with an estimate, a test plan and a fallback shows the judgement teams look for.

How should you learn about scaling?

Train several small GPT-style models of different sizes on the same data, plot validation loss against parameters and against tokens seen, and see the curves for yourself. Scaling laws and compute-optimal training are part of the advanced LLM engineering topics in Program Zero's generative AI and LLM course, alongside distributed training, modern architectures and alignment. Start with building a GPT from scratch.