Choosing the right evaluation metric matters as much as choosing the model. For classification, start from the confusion matrix: precision measures how many flagged cases are real, recall how many real cases were caught, F1 balances the two, and ROC-AUC and PR-AUC summarise performance across thresholds. Accuracy is misleading on imbalanced data. For regression, MAE, RMSE and R² answer different questions. The right metric reflects the real cost of each kind of mistake.
What is a confusion matrix?
For a yes/no classifier, every prediction falls into one of four cells. Suppose a fraud model checks 1,000 transactions, of which 50 are actually fraudulent:
| Predicted fraud | Predicted genuine | |
|---|---|---|
| Actually fraud (50) | True positives (TP) = 40 | False negatives (FN) = 10 |
| Actually genuine (950) | False positives (FP) = 60 | True negatives (TN) = 890 |
Every classification metric is a different way of summarising these four numbers.
How do you calculate the main metrics?
| Metric | Formula | Value here | Answers |
|---|---|---|---|
| Accuracy | (TP + TN) ÷ total | 930 ÷ 1,000 = 93.0% | How often is the model right overall? |
| Precision | TP ÷ (TP + FP) | 40 ÷ 100 = 40.0% | When it flags fraud, how often is it right? |
| Recall (sensitivity) | TP ÷ (TP + FN) | 40 ÷ 50 = 80.0% | What share of fraud did it catch? |
| Specificity | TN ÷ (TN + FP) | 890 ÷ 950 ≈ 93.7% | What share of genuine cases did it clear? |
| F1 score | 2 × P × R ÷ (P + R) | ≈ 53.3% | Balance of precision and recall |
Computed from the confusion matrix above.
Why is accuracy misleading here?
A model that labels every transaction "genuine" scores 95% accuracy (950 of 1,000 correct), higher than our model's 93%, while catching zero fraud. Whenever one class is rare, accuracy mostly measures how common the majority class is. That's why fraud, disease screening, defect detection, churn and many other real problems are judged on precision, recall and related metrics instead.
How do you choose between precision and recall?
Ask which mistake costs more:
| Problem | Costlier mistake | Prioritise |
|---|---|---|
| Disease screening | Missing a sick patient (FN) | Recall |
| Spam filtering | Hiding a real email (FP) | Precision |
| Fraud review queue | Both; limited reviewer capacity | Recall at a precision reviewers can handle |
| Search results | Irrelevant top results | Precision at the top (precision@k) |
Most classifiers output a score, and moving the decision threshold trades one for the other. Choosing the threshold is a business decision informed by metrics; see the threshold section of logistic regression explained.
What are ROC-AUC and PR-AUC?
The ROC curve plots recall (true positive rate) against the false positive rate across all thresholds; ROC-AUC is the area under it. It equals the probability that the model scores a random positive higher than a random negative: 0.5 is random guessing, 1.0 perfect ranking. The precision-recall curve plots precision against recall; its area (PR-AUC, or average precision) focuses on the positive class and is much more informative when positives are rare, because ROC-AUC can look excellent even when precision at useful recall levels is poor.
Which metrics apply to multi-class problems?
Compute precision, recall and F1 per class, then average. Macro averaging treats every class equally (good when small classes matter); weighted averaging weights by class size; micro averaging pools all decisions. Always inspect the full confusion matrix too: it shows which classes get confused with which, which often reveals labelling problems or genuinely similar categories.
How do you evaluate regression models?
| Metric | Use when |
|---|---|
| MAE | You want typical error in real units, robust to outliers |
| RMSE | Large errors are especially costly |
| R² | Comparing models on the same data |
| MAPE / sMAPE | Comparing across products or scales (avoid near-zero targets) |
| Quantile (pinball) loss | You need ranges, such as safety stock levels |
See linear regression explained for a worked example.
How do you make evaluation trustworthy?
- Hold out data the model never sees during training or tuning, and use cross-validation to estimate variability.
- Respect time: for time-dependent data, validate on later periods.
- Compare with baselines: majority class, simple rules, last year's model.
- Slice results by city, language, device or customer type; averages hide failures.
- Check calibration when probabilities drive decisions.
- Translate to business impact: "catches 80% of fraud with 60 false alarms per 1,000 transactions" is clearer than "F1 = 0.53".
How are metrics used for LLMs?
Language model systems add new metrics: perplexity for language modelling, BLEU and ROUGE for overlap with reference texts, exact-match and F1 for question answering, and rubric- or model-based grading for open-ended answers, plus faithfulness checks for retrieval systems. The principle is unchanged: pick metrics that reflect what users need, build a fixed test set, and track results over time. Read AI hallucinations explained for faithfulness in particular.
What do the metrics look like at different thresholds?
Suppose the same fraud model is run at three thresholds on the 1,000 transactions:
| Threshold | Flagged | Fraud caught (of 50) | Precision | Recall |
|---|---|---|---|---|
| High (0.8) | 30 | 25 | 83% | 50% |
| Medium (0.5) | 100 | 40 | 40% | 80% |
| Low (0.2) | 250 | 47 | 19% | 94% |
These numbers are illustrative, but the pattern is universal: as the threshold falls, recall rises and precision falls. If the fraud team can review 100 transactions a day, the medium threshold matches their capacity; if a missed fraud costs far more than a review, the low threshold may be worth hiring more reviewers for. Metrics don't make this decision, but they make the trade-off visible.
What other classification metrics are useful?
- Matthews correlation coefficient (MCC): uses all four confusion-matrix cells and stays informative on imbalanced data.
- Balanced accuracy: the average of recall on each class, which avoids accuracy's majority-class trap.
- Log loss: evaluates the probabilities themselves, punishing confident mistakes.
- Precision@k and recall@k: how good the top k results are, for ranking and recommendation.
- Cost-weighted error: assign a rupee cost to each error type and minimise the total, the most business-aligned metric of all.
How should you report metrics to non-technical stakeholders?
Translate every metric into the outcome people care about. Instead of "precision 40%, recall 80%", say "out of every 100 transactions we flag, 40 are real fraud, and we catch 8 out of every 10 frauds that happen." Show the trade-off at two or three candidate thresholds, with the workload and money involved at each. Include a simple baseline for comparison, state the time period and data the numbers come from, and mention where the model is weaker, such as certain cities or new customers. Clear, honest reporting builds the trust that gets models deployed, and protects you when results in production differ from the test set.
Finally, agree on the metric before modelling starts. Teams that decide afterwards are tempted to pick whichever number looks best, which is a quiet form of overfitting to the evaluation itself. Writing the chosen metric and target down at the start keeps everyone honest.
How should you practise?
Take any classifier you've built, compute the confusion matrix and every metric above by hand once, then with scikit-learn. Plot precision and recall against the threshold and pick a threshold for a stated business goal. Model evaluation and tuning are taught throughout Program Zero's machine learning course and revisited for LLMs in the generative AI course.