CI/CD for machine learning automates the path from a code or data change to a deployed model. Continuous integration runs on every change: linting, unit tests, data validation and a quick training smoke test. Continuous delivery then trains or retrains the model, evaluates it against agreed thresholds and the current production model, registers it, builds a container, and deploys it gradually. GitHub Actions can orchestrate all of this with workflow files in your repository.

Why do ML teams need CI/CD?

Manual ML releases are slow and risky: someone retrains on their laptop, copies a model file to a server, and hopes the preprocessing matches. Bugs slip in, models silently get worse, and nobody can say which code and data produced what's running. CI/CD replaces that with a repeatable, audited process. It also enables continuous training: as new data arrives, models are retrained and promoted only when they demonstrably improve.

What is different about ML compared with normal software?

AspectTraditional softwareMachine learning
What changesCodeCode, data and model
TestsUnit and integration testsPlus data validation and model-quality tests
Build outputAn application binary or imageAn image plus a trained model artifact
Release decisionTests passTests pass and evaluation beats thresholds
After releaseMonitor errors and latencyAlso monitor data drift and prediction quality

What should run on every pull request?

  • Formatting and linting (for example, Ruff or flake8) and type checks.
  • Unit tests for feature engineering, preprocessing and utility code.
  • Data validation on a sample: schema, ranges, missing values, allowed categories.
  • Training smoke test: train on a tiny dataset for one epoch to prove the pipeline runs end to end.
  • API tests for the serving code.

These checks should finish in minutes; slow pipelines get ignored.

How do evaluation gates work?

Before a model can be deployed, the pipeline evaluates it on a fixed validation or test set and compares with agreed rules: for example, F1 must be at least 0.70, must not be worse than the production model by more than 0.01, and recall for the minority class must not drop. Slice checks (by city, language or customer type) catch models that improve on average but fail a group. If any gate fails, the pipeline stops and reports why.

An example gate: candidate vs production model on three checks
Overall F1: production0.71
Overall F1: candidate0.74
Minority-class recall: production0.62
Minority-class recall: candidate0.64

Illustrative numbers. In this example both checks improve, so the gate passes.

What does a GitHub Actions setup look like?

Workflows are YAML files in .github/workflows/. A typical ML repository has three:

  1. ci.yml, triggered on pull requests: check out code, set up Python with dependency caching, install pinned requirements, run lint, tests, data validation on a sample and the training smoke test.
  2. train.yml, triggered on a schedule, on new data, or manually: pull the versioned data (for example with DVC), train on suitable compute (often a separate GPU runner or cloud job triggered from the workflow), log the run to MLflow, run the evaluation gates, and register the model if they pass.
  3. deploy.yml, triggered when a model is approved or on merge to main: build the Docker image with the model, push it to a registry, deploy to staging, run smoke tests, then promote to production gradually.

Secrets such as cloud credentials are stored as GitHub encrypted secrets and never committed. Using OpenID Connect to obtain short-lived cloud credentials avoids long-lived keys altogether.

How do you deploy safely?

StrategyHow it worksBenefit
Staging firstDeploy to a copy of production and testCatches environment problems
ShadowNew model sees real traffic; predictions only loggedReal-world comparison with no user impact
CanarySmall share of traffic goes to the new modelLimits blast radius
Blue-greenTwo environments; switch traffic at onceInstant rollback

Whatever the strategy, keep the previous model version deployable and make rollback a one-step action.

How does continuous training work?

Once the pipeline is automated, retraining can be triggered by a schedule (weekly), by data volume (10,000 new labelled examples), or by monitoring alerts (data drift detected). The same gates protect production: a retrained model is only promoted if it beats the current one. This closes the loop between deployment, monitoring and experiment tracking.

What does CI/CD look like for LLM applications?

For LLM apps, the "model" is often a combination of prompts, retrieval settings and an external model version. The same discipline applies: store prompts in the repository, run an evaluation suite of realistic questions on every change, check faithfulness and format, and block releases that regress. Our guides to prompt engineering and hallucinations describe what to test.

What mistakes should you avoid?

  • Running full training on every commit, making CI slow and expensive.
  • Evaluating on data that changes between runs, so comparisons are meaningless.
  • Unpinned dependencies that make training non-reproducible.
  • Committing secrets or large data files to Git.
  • No rollback plan.

What does data validation in CI check?

Data changes break models as often as code changes. A validation step checks a sample of new data against expectations: required columns exist with the right types; numeric values fall in plausible ranges (an age of 250 is an error); categorical columns contain only known values or flag new ones; missing-value rates haven't jumped; row counts are within normal bounds; and there are no duplicated IDs. Tools such as Great Expectations or Pandera express these rules as code. When a check fails, the pipeline stops before a model trains on broken data, and the report points straight at the problem.

How do you test ML code that involves randomness?

Fix random seeds in tests so results are repeatable. Test properties rather than exact numbers: the training loss should decrease over a few steps on a tiny dataset; predictions should have the right shape and lie in the valid range; a model trained on perfectly separable toy data should reach near-perfect accuracy. Keep a small golden set of inputs whose predictions are recorded, and flag large unexplained changes. These tests run in seconds and catch most pipeline bugs long before an expensive training job does.

What does a mature ML pipeline look like?

In a mature setup, a data scientist opens a pull request with a new feature. CI runs tests, validates a data sample and trains a tiny model in minutes. After review and merge, a scheduled training workflow retrains on the latest versioned data, logs the run, evaluates it against gates and slices, and registers it. A deployment workflow ships it to staging, then to a small share of production traffic, while dashboards compare it with the current model. If metrics hold for a day, it's promoted; if not, it's rolled back automatically. Nobody copies files by hand, and every step is recorded. Getting there takes time, so build it incrementally: tests first, then automated training, then gated deployment.

How do you build this skill?

Take one ML project and add a CI workflow with tests and a smoke-train step, then a deploy workflow that builds and pushes a Docker image. Add an evaluation gate last. CI/CD with GitHub Actions, Docker, Kubernetes, MLflow, DVC and monitoring are taught in Program Zero's MLOps and cloud deployment course.