Principal component analysis (PCA) reduces the number of features in a dataset while keeping as much of its variation as possible. It finds new axes, called principal components, ordered so the first captures the most variance, the second the most of what remains, and so on. Keeping the first few components gives a compact representation that's useful for visualisation, noise reduction and faster modelling.
Why reduce dimensions at all?
Real datasets often have dozens or hundreds of features, many of them correlated: height and weight, several test scores that move together, or hundreds of pixel values in an image. Too many features make data hard to visualise, slow some algorithms down, increase overfitting risk and dilute distances (the "curse of dimensionality"). Dimensionality reduction keeps the signal and drops redundancy. PCA is the classic, fast, well-understood way to do it.
What is the intuition behind PCA?
Imagine a cloud of points shaped like a long, thin cigar tilted in 3D space. Describing each point with x, y and z works, but most of the variation lies along the cigar's length. If you rotate your axes so that the first axis runs along the cigar, the second across its width and the third through its thickness, the first coordinate alone tells you most of what distinguishes points. PCA finds exactly that rotation for data in any number of dimensions, then lets you drop the axes that carry little variation.
How does PCA work, step by step?
- Standardise each feature to mean 0 and standard deviation 1 (in most cases).
- Compute the covariance matrix, describing how features vary together.
- Find its eigenvectors and eigenvalues (in practice, via singular value decomposition). Eigenvectors are the principal component directions; eigenvalues measure the variance along each.
- Sort components by eigenvalue, largest first.
- Project the data onto the top k components to get the reduced dataset.
Each component is a weighted combination of the original features, and the weights (called loadings) show which features drive it. Components are uncorrelated with each other, a useful property for some downstream models.
How do you choose how many components to keep?
Look at the explained variance ratio of each component and the cumulative total:
Illustrative values showing a typical shape; your data will differ. Plot this curve for your own dataset.
Common rules: keep enough components for 90–95% of variance; look for the point where the curve flattens; or treat k as a hyperparameter and choose the value that gives the best validation performance in your downstream model. For visualisation, you simply use 2 or 3.
What is PCA used for?
| Use | How PCA helps |
|---|---|
| Visualisation | Project high-dimensional data to 2D to see clusters and outliers |
| Preprocessing | Fewer, uncorrelated features for faster or more stable models |
| Noise reduction | Dropping low-variance components removes some noise |
| Compression | Store data or images with fewer numbers |
| Multicollinearity | Replace correlated predictors in regression with components |
| Exploring embeddings | Inspect structure in word or sentence embeddings |
Why does scaling matter so much?
PCA chases variance. If one feature is income in rupees (variance in the millions) and another is age (variance in the tens), the first component will essentially be "income" no matter what the data's real structure is. Standardising puts features on equal footing. The exception is when all features share the same units and their raw variances are meaningful, such as pixel intensities.
What are PCA's limitations?
- Linear only: PCA finds straight-line directions. Curved structures need methods such as kernel PCA, t-SNE, UMAP or autoencoders.
- Variance isn't always importance: the direction with most variance may be irrelevant to your target, and a small-variance direction may be the predictive one.
- Harder interpretation: components mix original features, which makes explanations less direct.
- Sensitive to outliers: extreme points can tilt components.
How does PCA compare with t-SNE and UMAP?
| Method | Preserves | Best for | Caveat |
|---|---|---|---|
| PCA | Global variance, linear structure | Preprocessing, quick overview | Misses non-linear structure |
| t-SNE | Local neighbourhoods | Visualising clusters | Distances between clusters aren't meaningful |
| UMAP | Local and some global structure | Visualisation, sometimes preprocessing | Sensitive to settings |
A common workflow is PCA to around 50 dimensions first, then t-SNE or UMAP to 2D for plotting.
What mistakes should you avoid?
- Fitting PCA on all data before splitting: fit on training data only, then apply the same transformation to validation and test data, or you leak information.
- Forgetting to scale.
- Assuming fewer features always helps: tree models often do better on original features.
- Over-interpreting component names: check loadings before calling a component "affordability".
How is PCA used on images?
A classic demonstration uses face images. Each 64 × 64 grayscale face is a vector of 4,096 pixel values. PCA on a collection of faces finds components, often called "eigenfaces", that capture lighting direction, face shape and common features. Faces can be reconstructed reasonably well from a few dozen components instead of thousands of pixels, and the compressed versions can feed a simple classifier. Modern face systems use deep networks instead, but the example shows clearly what PCA does: it finds a compact basis in which the data can be described with far fewer numbers.
How does PCA relate to other methods?
PCA is closely related to singular value decomposition (SVD), which is how it's usually computed and which also powers classic recommendation systems and latent semantic analysis of text. An autoencoder, a neural network trained to compress and reconstruct its input, learns a non-linear version of the same idea; a linear autoencoder with squared-error loss learns the same subspace as PCA. Low-rank ideas from the same family appear in LLM fine-tuning too: LoRA adapts a model with low-rank weight updates, built on the observation that useful changes often live in a small number of directions.
Understanding PCA's linear algebra therefore pays off well beyond dimensionality reduction. It reappears in recommendation systems, compression, embeddings analysis and parameter-efficient fine-tuning, which is why it's worth working through one small example by hand.
How is PCA asked about in interviews?
Typical questions: what do eigenvectors and eigenvalues represent in PCA, why should you standardise first, how do you choose the number of components, is PCA supervised or unsupervised (unsupervised: it ignores the target), and when would you avoid it (when interpretability of original features matters, when relationships are non-linear, or when the predictive signal sits in low-variance directions). A strong candidate also explains why PCA must be fitted on training data only, and can sketch the explained-variance curve from memory.
Practising these answers out loud, with one concrete dataset in mind, turns abstract linear algebra into an explanation an interviewer can follow, and shows that you have actually used PCA rather than only read about it.
How should you practise?
Run PCA on a dataset of correlated features (exam scores, sensor readings or images of digits), plot the cumulative explained variance, visualise the first two components coloured by a known label, and compare a model's accuracy with and without PCA. PCA appears alongside k-means in the unsupervised learning part of Program Zero's machine learning course, and the linear algebra behind it is introduced when you need it.