Cross-validation estimates how well a model will perform on data it has never seen by training and testing it several times on different splits of the training data and averaging the results. Hyperparameter tuning uses those cross-validated scores to choose settings such as tree depth or learning rate. Done properly, with a final untouched test set and no leakage, they tell you the truth about a model; done carelessly, they produce impressive numbers that collapse in production.
Why isn't a single train/test split enough?
With one split, your score depends partly on luck: which examples happened to land in the test set. On small datasets this luck can swing results by several percentage points, enough to make a worse model look better. And if you try many models and settings against the same test set, you gradually fit your choices to that particular test set, so its score stops being an honest estimate. Cross-validation reduces the luck, and a separate final test set protects you from the second problem.
What are the train, validation and test sets for?
| Set | Used for | Touched how often |
|---|---|---|
| Training | Learning parameters (weights, splits) | Constantly |
| Validation (or CV folds) | Choosing models, features and hyperparameters | Many times |
| Test | Final, honest estimate of performance | Once, at the end |
A common split is 70/15/15 or 80/10/10, but with cross-validation you can merge training and validation and rotate through folds instead.
How does k-fold cross-validation work?
Split the training data into k equal folds (say 5). Train on folds 2–5 and evaluate on fold 1; train on folds 1, 3, 4, 5 and evaluate on fold 2; and so on, until every fold has been the validation set once. Average the five scores, and look at their spread: a model scoring 0.82 ± 0.01 is more trustworthy than one scoring 0.83 ± 0.06.
Arithmetic: (k − 1) ÷ k. More folds train on more data per round but need k model fits.
Which cross-validation scheme fits your data?
| Scheme | Use when |
|---|---|
| K-fold | Independent examples, balanced classes |
| Stratified k-fold | Classification, especially imbalanced: each fold keeps the class ratio |
| Group k-fold | Several rows per person, store or patient: keep each group in one fold |
| Time-series split | Data ordered in time: always validate on later periods |
| Leave-one-out | Very small datasets (expensive) |
| Nested CV | You need an unbiased estimate while also tuning |
Group and time-based splits matter more than people expect. If the same customer appears in both training and validation, the model can recognise the customer rather than learning general patterns, and scores will be inflated.
What are hyperparameters?
Parameters are learned during training: a regression's weights, a tree's split thresholds, a neural network's weights. Hyperparameters are choices you make before training that control how learning happens: tree depth, number of trees, learning rate, regularisation strength, number of neighbours in k-NN, batch size and network width. They can't be learned by the same training process, so you choose them by comparing validation performance across candidates.
How do grid search, random search and Bayesian optimisation compare?
| Method | How it works | Strength | Weakness |
|---|---|---|---|
| Grid search | Tries every combination of listed values | Simple, exhaustive on small grids | Explodes with more hyperparameters |
| Random search | Samples combinations at random | Covers important dimensions efficiently | May miss narrow optima |
| Bayesian optimisation (e.g. Optuna) | Uses past results to choose promising next trials | Fewer trials for expensive models | More setup |
| Early-stopping / successive halving | Stops weak trials early | Saves compute on many trials | Needs incremental training |
A grid of 5 values for each of 4 hyperparameters is already 625 combinations; with 5-fold CV that's 3,125 model fits. Random search with 60 trials often finds a result just as good, because usually only one or two hyperparameters matter much.
What is data leakage, and how does it sneak in?
Leakage means information from validation or test data influences training, so scores look better than real-world performance will be. Common forms:
- Preprocessing before splitting: scaling, imputing or selecting features using the whole dataset. Fit these steps on training folds only; scikit-learn
Pipelineobjects inside cross-validation do this correctly. - Target leakage: features that are only known after the outcome (for example, "refund issued" when predicting complaints).
- Duplicates across splits: near-identical rows in both training and validation.
- Time leakage: using future data to predict the past.
- Tuning on the test set: repeatedly checking the test score and adjusting.
What does a sound workflow look like?
- Split off a test set first (respecting time and groups) and lock it away.
- Build a pipeline: preprocessing steps plus the model.
- Use cross-validation on the remaining data to compare models and tune hyperparameters.
- Refit the chosen pipeline on all non-test data.
- Evaluate once on the test set and report that number, with its uncertainty.
- After deployment, monitor real performance; it is the ultimate test.
How does this apply to deep learning?
Neural networks are expensive to train, so full k-fold CV is often impractical. Teams typically use a fixed validation set with early stopping and tune learning rate, batch size, regularisation and architecture with random or Bayesian search, often stopping weak trials early. For fine-tuning LLMs, the same discipline applies: a held-out evaluation set that never touches training, and no repeated peeking at the final test set.
How do you read cross-validation results?
Look at the mean and the spread. If two models' averages differ by less than the fold-to-fold variation, the difference may be noise, and the simpler or cheaper model is usually the better choice. Check per-fold results for a fold that behaves very differently, which can reveal a data issue such as a cluster of mislabelled rows or a period when data was collected differently. Compare the training score with the validation score too: a large gap points to overfitting, while both being low points to underfitting.
What mistakes do practitioners make?
- Reporting the best validation score after hundreds of trials as if it were a test score.
- Using random k-fold on time series.
- Tuning dozens of hyperparameters on a small dataset, which overfits the validation folds.
- Comparing models evaluated on different splits.
- Forgetting to fix random seeds when comparing results.
What is nested cross-validation?
When you tune hyperparameters with cross-validation and then report that same cross-validated score, the estimate is optimistic: you picked the settings that happened to score best on those folds. Nested cross-validation fixes this with two loops. The inner loop tunes hyperparameters on part of the data; the outer loop evaluates the whole tuning procedure on folds the inner loop never saw. It's expensive, but it gives an honest estimate when data is too scarce to set aside a separate test set, which is common in medical and scientific datasets.
How should you practise?
Take a dataset, deliberately leak information by scaling before splitting and by including a leaky feature, watch the score jump, then fix both and see the honest number. Next, tune a gradient-boosting model with random search inside a pipeline and compare against a grid. Model evaluation and tuning are a dedicated topic in Program Zero's machine learning course; read evaluation metrics explained and XGBoost explained alongside this guide.