A decision tree predicts by asking a learned sequence of yes/no questions about the input ("is attendance above 75%?", "are weekly practice hours above 5?") until it reaches a leaf with a prediction. A random forest trains hundreds of trees on random samples of rows and features and averages their answers, which keeps the trees' flexibility while removing most of their tendency to overfit. Both handle numbers and categories, need no feature scaling and are among the strongest models for tabular data.

How does a decision tree make predictions?

Picture predicting whether a learner will finish a course. The tree's first question might be "attendance above 70%?". Learners who answer yes go left, where the next question is "submitted at least 8 assignments?"; those who answer no go right, where the next question is "works full time?". After a few questions, every learner lands in a leaf, and the leaf predicts the most common outcome (classification) or the average value (regression) of the training examples that ended up there. The model is literally a flowchart, which is why trees are so easy to explain to non-technical colleagues.

How does a tree choose its questions?

At each node, the algorithm tries possible splits (every feature, and many threshold values for numeric features) and picks the one that makes the resulting groups purest. Purity is measured with:

  • Gini impurity: the chance that two random examples from the group have different labels. 0 means perfectly pure.
  • Entropy / information gain: how much uncertainty about the label the split removes.
  • Variance reduction or squared error for regression trees.

Example: a node holds 50 finishers and 50 dropouts, Gini = 1 − (0.5² + 0.5²) = 0.5. A split producing groups of (40 finished, 10 dropped) and (10 finished, 40 dropped) leaves each group with Gini = 1 − (0.8² + 0.2²) = 0.32, a clear improvement. The tree greedily repeats this until a stopping rule is reached.

Gini impurity of a two-class group as it becomes purer
50% / 50%0.50
70% / 30%0.42
80% / 20%0.32
90% / 10%0.18
100% / 0%0.00

Computed with Gini = 1 − p² − (1 − p)².

Why do single trees overfit?

Left unchecked, a tree keeps splitting until every leaf is pure, memorising the training data including its noise: a leaf might exist just for one unusual learner. Such a tree scores perfectly on training data and poorly on new data, and small changes in the data can produce a completely different tree (high variance). The cures are limits on growth: maximum depth, minimum examples per leaf, minimum impurity decrease, or pruning branches that don't improve validation performance. Tuning these with cross-validation is essential.

How does a random forest fix this?

A random forest builds many deep trees and makes them different from one another in two ways:

  1. Bagging (bootstrap aggregating): each tree trains on a random sample of rows drawn with replacement, so each sees a slightly different dataset.
  2. Feature randomness: at each split, a tree may only consider a random subset of features (commonly about the square root of the number of features for classification).

Each tree overfits in its own way, but their errors are partly independent, so averaging hundreds of them cancels much of the noise while keeping the signal. A useful by-product is the out-of-bag estimate: each row was left out of roughly a third of the trees, so those trees can be used to estimate performance without a separate validation set.

How do trees and forests compare?

AspectSingle decision treeRandom forest
InterpretabilityExcellent: a readable flowchartLower: hundreds of trees
AccuracyModerateUsually much higher
Overfitting riskHigh without limitsLow
Training speedFastSlower, but parallelises well
Tuning neededDepth and leaf limits matter a lotWorks well with defaults
Feature scalingNot neededNot needed

What is feature importance, and can you trust it?

Forests report how much each feature reduced impurity across all trees. This "impurity importance" is quick but biased towards features with many possible split points, such as IDs or continuous values. Permutation importance is more reliable: shuffle one feature in the validation data and measure how much performance drops. For explaining individual predictions, SHAP values are widely used. Treat any importance as association within your model, not proof that the feature causes the outcome.

Which hyperparameters matter?

HyperparameterEffectTypical starting point
n_estimatorsNumber of trees; more is better until it plateaus200–500
max_depthLimits tree sizeNone (forests) or 5–15 (single trees)
min_samples_leafSmooths predictions; reduces overfitting1–10
max_featuresFeatures tried per split; lower adds diversity"sqrt" for classification
class_weightHandles imbalanced classes"balanced" when the positive class is rare

Where are trees and forests used?

Tree models dominate tabular business problems: credit risk scoring, churn prediction, fraud flags, demand estimation, lead scoring and many government and healthcare datasets. They cope with mixed feature types, missing values (in many implementations) and non-linear interactions without manual feature engineering. They are weaker on images, audio and raw text, where deep learning wins, and they cannot extrapolate beyond the range of target values seen in training, which matters for forecasting growth.

What does a practical workflow look like?

  1. Start with a simple baseline, such as logistic regression.
  2. Train a shallow decision tree and read it: does it make sense to a domain expert? Odd splits often reveal data problems.
  3. Train a random forest with defaults; compare validation metrics with the baseline.
  4. Tune a few hyperparameters with cross-validation.
  5. Check permutation importance and error cases.
  6. Compare with gradient boosting before choosing.

What mistakes should you avoid?

  • Leaky features: trees find leaks brilliantly, such as a "refund issued" column predicting churn perfectly. Remove anything not known at prediction time.
  • Judging by training accuracy: a deep tree is always near 100% on training data.
  • Over-trusting impurity importance for decisions about the business.
  • Ignoring imbalance: use the right metrics and class weights.

How do trees handle different kinds of features?

For numeric features, a tree tests thresholds such as "age ≤ 24.5". For categorical features, implementations differ: scikit-learn needs categories encoded as numbers first (one-hot or ordinal encoding), while LightGBM and CatBoost can split on categories directly. Missing values are handled natively by many modern implementations, which learn whether missing cases should go left or right at each split. Because splits only care about order, trees are unaffected by monotonic transformations such as logs, and they ignore the units features are measured in, one reason they need so little preprocessing compared with linear models or neural networks.

How are trees asked about in interviews?

Expect to explain how a split is chosen (Gini or entropy), why deep trees overfit, how bagging and feature randomness reduce variance, what out-of-bag error is, and why tree models don't need scaling. A good answer also compares random forests with gradient boosting and mentions the risk of misleading impurity-based feature importance. Being able to compute a Gini value for a small example on a whiteboard, as above, shows real understanding quickly.

How do you learn this properly?

Implement a tiny decision tree with Gini splits in Python once, then use scikit-learn's DecisionTreeClassifier and RandomForestClassifier on a real dataset and compare them. Trees, ensembles, evaluation and tuning are core topics in Program Zero's machine learning course, which ends with a deployed end-to-end project; see ML algorithms for beginners for how they fit with other methods.