Linear regression predicts a number from one or more inputs by fitting a straight line (or a flat plane) through the data: prediction = intercept + weight₁ × input₁ + weight₂ × input₂ + …. Training chooses the weights that minimise the squared prediction errors. It is fast, easy to interpret and the best first model to learn, because every idea in it (loss, training, overfitting, evaluation) reappears in deep learning and LLMs.
What problem does linear regression solve?
Any time you want to predict a quantity from other measurements, linear regression is the natural starting point: a flat's rent from its area and distance to a metro station, a delivery's time from distance and order size, monthly electricity use from temperature, or a student's final score from attendance and practice hours. It assumes the target changes by a roughly constant amount for each unit change in each input. That sounds restrictive, but with a little feature engineering it handles a surprising range of problems, and even when it isn't the final model it's the baseline every other model must beat.
What does the equation mean?
With one input, the model is ŷ = b + w·x. b (the intercept) is the prediction when x is zero; w (the slope or weight) is how much the prediction rises for each one-unit increase in x. With several inputs it becomes ŷ = b + w₁x₁ + w₂x₂ + … + wₙxₙ, and each weight tells you the effect of that input while holding the others constant. That interpretability is the main reason linear models are still used heavily in finance, economics, policy and medicine.
How does the model learn the weights?
For each training example, the error (residual) is actual minus predicted. Linear regression minimises the mean squared error (MSE): the average of the squared residuals. Squaring makes all errors positive and punishes large errors heavily. There are two ways to find the best weights:
- Closed form (ordinary least squares): a formula from linear algebra (the normal equation) gives the exact answer in one step. Fast for small and medium problems.
- Gradient descent: start with random weights, compute how the error changes as each weight changes (the gradient), take a small step downhill, and repeat. Slower for this model, but it scales to huge datasets and is exactly how neural networks learn, so it's worth understanding here first.
What does a worked example look like?
Five small flats, area in hundreds of square feet (x) and monthly rent in thousands of rupees (y): (5, 12), (6, 14), (7, 15), (8, 18), (10, 21). The mean x is 7.2 and the mean y is 16. Least squares gives slope w = Σ(x−x̄)(y−ȳ) ÷ Σ(x−x̄)² = 27 ÷ 14.8 ≈ 1.82 and intercept b = ȳ − w·x̄ ≈ 16 − 1.82 × 7.2 ≈ 2.86. The model: rent ≈ 2.86 + 1.82 × area. Each extra 100 sq ft adds about ₹1,820 a month to the predicted rent, and a 900 sq ft flat is predicted at about ₹19,300.
| Area (×100 sq ft) | Actual rent (₹ thousand) | Predicted | Error |
|---|---|---|---|
| 5 | 12 | 11.99 | +0.01 |
| 6 | 14 | 13.81 | +0.19 |
| 7 | 15 | 15.64 | −0.64 |
| 8 | 18 | 17.46 | +0.54 |
| 10 | 21 | 21.11 | −0.11 |
Computed from the worked example above.
The errors are small and roughly balanced around zero, which is what a good fit looks like. With real data you'd hold some flats back as a test set and check the error there, not only on the five used for training.
How do you evaluate a regression model?
| Metric | Meaning | Good for |
|---|---|---|
| MAE (mean absolute error) | Average size of errors, in the target's units | Easy to explain: "off by ₹800 on average" |
| RMSE (root mean squared error) | Like MAE, but punishes big errors more | When large mistakes are costly |
| R² | Share of variance explained | Comparing models on the same data |
| MAPE | Average percentage error | Comparing across scales (careful near zero) |
Always compute metrics on data the model didn't see during training, ideally with cross-validation. A model with R² of 0.95 on training data and 0.60 on test data is overfitting.
What assumptions does linear regression make?
- Linearity: the target changes roughly linearly with each input. Check with scatter plots and residual plots.
- Independent errors: one example's error doesn't predict another's. Time series often break this.
- Constant spread of errors across the range of predictions (homoscedasticity).
- No extreme multicollinearity: inputs that are near-copies of each other make weights unstable and hard to interpret.
- Roughly normal errors, mainly needed for confidence intervals and p-values, not for prediction itself.
For pure prediction, a model can be useful even when assumptions bend. For explaining effects ("each year of experience adds X"), violated assumptions can make conclusions wrong.
How do you handle non-linear patterns and categories?
Linear regression is linear in its weights, not necessarily in the raw inputs. You can add squared terms (area²) to capture curves, take logs of skewed variables such as income or price, and add interaction terms (area × city tier) when effects depend on each other. Categories such as city or property type become numbers through one-hot encoding: one 0/1 column per category. These tricks are part of feature engineering, and they often matter more than switching to a fancier algorithm.
What are regularisation, Ridge and Lasso?
With many features, plain least squares can overfit: weights grow large to fit noise. Regularisation adds a penalty for large weights. Ridge (L2) shrinks all weights smoothly and helps when features are correlated. Lasso (L1) can shrink some weights exactly to zero, effectively selecting features. Elastic Net mixes both. The penalty strength is a hyperparameter you tune with cross-validation. The same idea, called weight decay, is used when training neural networks.
What are the common mistakes?
- Extrapolating: a model trained on flats of 500–1,000 sq ft says little about a 5,000 sq ft villa.
- Reading weights as causes: regression shows association; causal claims need careful design.
- Unscaled features with regularisation: penalties treat all weights equally, so scale features first.
- Leaking the target: including a feature computed from the answer makes test scores meaningless.
- Letting outliers dominate: squared error is sensitive to them; investigate, or use robust alternatives.
How does linear regression connect to deep learning?
A single neuron in a neural network computes exactly a weighted sum plus a bias, then applies an activation function. Remove the activation and a one-neuron network is linear regression. Training it with gradient descent on squared error is the same process used to train huge networks, just at a tiny scale. That is why Program Zero teaches linear models first: backpropagation and neural networks are much easier once this is second nature.
How should you practise?
- Implement linear regression with gradient descent in NumPy, then compare with scikit-learn's
LinearRegression. - Fit a model to a real dataset (rents, used-car prices, crop yields), report MAE on a test set and plot residuals.
- Add a log transform or interaction and see whether test error improves.
- Try Ridge and Lasso and compare which features Lasso drops.
Next, read logistic regression explained for classification. Linear models, trees, ensembles and evaluation make up Program Zero's machine learning course, which ends in a deployed end-to-end project.