To deploy a machine learning model with FastAPI: save the trained model (and its preprocessing) as one artifact, load it once when the app starts, define request and response schemas with Pydantic, expose a /predict endpoint, add a health check, test it, package everything in a Docker image, run it behind a production server, and monitor latency, errors and prediction drift. This turns a notebook result into something other software and people can actually use.
Why does deployment matter?
A model in a notebook helps nobody. Deployment makes predictions available to a website, a mobile app, an internal tool or another service, reliably and at the speed users need. It is also where many ML projects fail: preprocessing differs between training and serving, library versions drift, inputs arrive in unexpected formats, or the model is too slow. A clean deployment pattern avoids most of these problems and is one of the most valued skills for ML engineers.
Step 1: how should you save the model?
Save the whole pipeline, preprocessing included, not just the estimator. In scikit-learn, wrap scalers, encoders and the model in a Pipeline and save it with joblib. For PyTorch, save the state_dict plus the code that defines the model, or export to a format such as TorchScript or ONNX. Record library versions and a model version string, and store the artifact in a registry or object storage (see MLflow explained), not inside your source code repository if it's large.
Step 2: how do you define the API?
Pydantic models describe exactly what a request must contain, and FastAPI validates every request automatically, returning a clear 422 error for bad input:
- A request model such as
LoanApplicationwith typed fields (income: float,city: str,age: intwith a range). - A response model with the prediction, a probability and the model version.
- A
POST /predictendpoint that converts the request to the model's input format, calls the model and returns the response. - A
GET /healthendpoint that confirms the model is loaded.
FastAPI generates interactive documentation at /docs, where you and your frontend colleagues can try requests in the browser.
Step 3: how do you load the model efficiently?
Load the model once at startup (using FastAPI's lifespan event) and keep it in memory. Loading per request can add seconds of latency. If prediction is CPU-heavy and synchronous, define the endpoint as a normal (non-async) function so FastAPI runs it in a thread pool, or run several worker processes, so one slow prediction doesn't block other requests.
Step 4: how do you test the service?
- Unit tests for preprocessing functions.
- API tests with FastAPI's
TestClient: valid input returns 200 and a sensible prediction; invalid input returns 422. - Consistency tests: predictions from the API match the notebook's predictions for the same inputs, which catches training/serving skew.
- Load tests with a tool such as Locust to measure latency under realistic traffic.
Step 5: how do you containerise it?
A Dockerfile starts from a slim Python base image, installs pinned dependencies from requirements.txt, copies the code and model artifact, and starts the server with Uvicorn (or Gunicorn managing Uvicorn workers) on a port. Build the image, run it locally, hit /health and /predict, then push it to a container registry. The same image runs on any cloud: a managed container service, Kubernetes, or a plain virtual machine.
How fast does it need to be?
An example latency budget for planning, not a measurement. Measure your own service with load tests.
Set a latency target with the product team, measure each stage, and optimise the largest one first, often model size (a smaller model, quantization, ONNX Runtime) or a slow feature lookup.
How do you scale a model API?
| Need | Approach |
|---|---|
| More requests per second | More worker processes or container replicas behind a load balancer |
| Lower latency | Smaller or quantised model, faster runtime, caching repeated inputs |
| GPU efficiency | Batch requests; use a model server such as NVIDIA Triton |
| Large LLMs | Specialised serving engines (for example vLLM) with continuous batching |
| Slow, non-urgent jobs | Queue them and process in background workers |
What should you monitor after launch?
- Service health: request rate, error rate, latency percentiles (p50, p95, p99).
- Inputs: missing fields, new categories, and shifts in feature distributions compared with training data.
- Outputs: the distribution of predictions; a sudden change often signals a data problem.
- Outcomes: when true labels arrive later (did the customer churn?), compute real accuracy over time.
Log requests and predictions (without sensitive personal data, following India's DPDP Act) so you can investigate problems and build the next training set.
What about security?
Require authentication (an API key or token) for anything not meant to be public, validate and bound every input, rate-limit to prevent abuse and runaway costs, keep secrets in environment variables or a secrets manager, and don't return internal error details. For models handling personal data, minimise what you store and for how long.
How do you roll out a new model version safely?
Never swap models blindly. Common strategies: shadow deployment, where the new model receives copies of real requests and its predictions are logged but not used; canary release, where a small share of traffic goes to the new model while you watch errors and metrics; and A/B testing, where two versions serve different users and you compare business outcomes. Keep the previous version ready for instant rollback, and include the model version in every response and log line so any problem can be traced to the exact model that produced it.
Should you choose online or batch prediction?
Not every model needs a real-time API. Online prediction answers each request immediately, which suits fraud checks, recommendations on a page or a chatbot. Batch prediction scores many records on a schedule, for example every night, and stores the results in a database, which suits churn scores for a weekly campaign or demand forecasts for tomorrow. Batch is simpler and cheaper to run and easier to monitor; online is needed when the answer depends on information available only at request time. Many systems combine both: nightly batch scores for most decisions, plus an online model for the few that need fresh data.
What mistakes do people make when deploying models?
- Training/serving skew: preprocessing reimplemented differently in the API. Ship the same pipeline object.
- Unpinned dependencies, so a library update changes predictions or breaks loading.
- No input limits: huge payloads or extreme values crash the service.
- No version in responses, so problems can't be traced to a model.
- No monitoring until users complain.
How do you practise?
Deploy a model you've trained: build the FastAPI service, write tests, containerise it, run it on a free cloud tier, and add a simple dashboard of latency and prediction counts. Model serving with FastAPI and Triton, Docker, Kubernetes, CI/CD and monitoring are taught in Program Zero's MLOps and cloud deployment course, whose project deploys an LLM and chatbot with a full CI/CD pipeline. Read building REST APIs for general API design.