Feature engineering is turning raw data into inputs a model can learn from: encoding categories, scaling and transforming numbers, extracting parts of dates, aggregating histories (counts, averages, recency), creating ratios and interactions, and representing text. For tabular data it is often the biggest source of improvement, more than switching algorithms. Every feature must be available at prediction time and computed without peeking at validation data.

Why does feature engineering matter?

A model can only learn from what you give it. Raw data often hides the useful signal: a timestamp means little, but "hour of day" and "is weekend" may predict delivery delays; a list of transactions means little, but "number of transactions in the last 7 days" may predict fraud. Domain knowledge turned into features lets simple models perform well, and makes complex ones more accurate and stable.

How do you encode categorical variables?

MethodHow it worksGood forWatch out
One-hotOne 0/1 column per categoryFew categories; linear modelsExplodes with many categories
OrdinalCategories mapped to ordered integersTruly ordered values (low/medium/high)Implies false order if misused
FrequencyReplace category with its count or shareMany categories; treesDifferent categories can collide
Target encodingReplace with the target's mean for that categoryHigh-cardinality columns (pin codes)Leakage unless computed out-of-fold
Native handlingLibraries like CatBoost and LightGBM split on categoriesTabular boostingLibrary-specific settings

Group rare categories into "Other" and decide how to handle categories that appear only after deployment.

How should numeric features be transformed?

  • Scaling: standardisation or min-max scaling for linear models, k-NN, SVMs and neural networks; trees don't need it.
  • Log transforms: for skewed values such as income, prices or counts, which reduces the influence of extremes.
  • Binning: age groups or score bands, when the relationship is step-like or for interpretability.
  • Clipping: capping extreme values at a percentile to limit outlier influence.
  • Ratios and differences: loan-to-income ratio, price per square foot, change since last month.

What can you extract from dates and times?

Day of week, hour, month, quarter, is-weekend, is-holiday (Indian national and regional holidays matter for many businesses), days until or since an event, and season. For cyclic values such as hour or month, sine and cosine encodings tell a model that 23:00 is close to 00:00. Time since the last event (recency) is often one of the strongest features in customer and behaviour models.

Why are aggregate features so powerful?

When data has a history per customer, learner or device, aggregates summarise behaviour: counts, sums, averages, minimums, maximums, trends and recency over windows such as 7, 30 and 90 days. Examples: logins in the last 7 days, average assignment score in the last month, days since last payment, number of distinct cities transacted from today.

Example: how many candidate features a few aggregates create
1 metric × 1 window × 1 statistic1
3 metrics × 3 windows × 1 statistic9
3 metrics × 3 windows × 4 statistics36

Arithmetic. Candidate features multiply quickly, so evaluate them carefully rather than adding everything.

The crucial rule: compute aggregates using only data available before the prediction time ("point-in-time correctness"). Using a month that includes the outcome leaks the future.

How do you build features from text?

Simple, strong options include length, word counts, presence of keywords and TF-IDF vectors for linear models. Modern options use embeddings from pretrained language models, which capture meaning, and can be fed to gradient-boosting models alongside tabular features. For short text such as product titles or support-ticket subjects, embeddings often add signal that keyword features miss.

What are interaction features?

Interactions capture effects that depend on combinations: area × city tier for rent, discount × category for sales. Tree models find many interactions automatically; linear models need them explicitly. Add interactions guided by domain knowledge rather than every possible pair, which creates noise and overfitting.

How do you choose which features to keep?

  • Validation-based: add or remove features and compare cross-validated scores on the same folds.
  • Importance-based: permutation importance or SHAP to find features that matter, and suspiciously dominant ones.
  • Regularisation: L1 can drop unhelpful features in linear models.
  • Simplicity: fewer features mean less to compute, monitor and explain in production.

How do you avoid leakage?

Fit encoders, scalers and imputers on training folds only, ideally inside a pipeline. Compute target encodings out-of-fold. Build historical aggregates with strict time cut-offs. Remove features created after the outcome (a "cancellation reason" when predicting cancellations). If one feature makes a model nearly perfect, investigate it before celebrating. Our guide to cross-validation and leakage covers this in detail.

What does feature engineering look like in production?

Features used in training must be computed identically at prediction time, often in real time. Teams therefore write feature logic once and reuse it, sometimes through a feature store that serves the same features for training and online prediction. Monitoring feature distributions after deployment catches broken pipelines and drift early.

What does a worked example look like?

Predicting whether a learner will submit next week's assignment from a raw activity log (one row per login and per submission). Useful features include: logins in the last 7 and 30 days, days since last login, number of assignments submitted so far, average score so far, share submitted late, whether they attended the last live class, day of week they usually study, and whether they work full time (from the profile). Each is computed only from data before the prediction date. A gradient-boosting model on these features will typically far outperform one given only raw profile fields, because the features express behaviour that the raw log hides.

How do features differ across algorithms?

Model typeNeeds scaling?Needs explicit interactions?Categorical handling
Linear and logistic regressionYes (with regularisation)YesOne-hot or target encoding
Tree ensemblesNoMostly learned automaticallyEncodings or native support
k-NN, SVMYesKernel or manualOne-hot
Neural networksYesLearnedEmbeddings for many categories

Knowing these differences stops you wasting time scaling features for a random forest, or wondering why a linear model ignores an obvious interaction.

Can AI help with feature engineering?

Automated tools can generate many candidate features (aggregates, lags, combinations) and select useful ones, and LLM assistants can suggest feature ideas from a description of the data. They speed up exploration, but they can also generate leaky or meaningless features at scale. Use them to propose candidates, then validate every feature with the same discipline: is it available at prediction time, does it improve validation performance across folds, and can you explain why it should help? Domain understanding remains the best source of strong features, especially for problems specific to Indian markets and customers, such as festival seasons, regional languages, cash-on-delivery habits or monsoon-driven demand that generic tools know nothing about.

How many features is too many?

There's no fixed number. Watch for validation scores that stop improving or start falling as you add features, and for features nobody can explain. Each feature also has a production cost: it must be computed, monitored and kept correct.

How should you practise?

Take a tabular dataset, build a baseline with raw features, then add one family of engineered features at a time (dates, aggregates, ratios, encodings), recording the validation score after each. Feature engineering, model evaluation and tuning are part of Program Zero's machine learning course; read the EDA guide first, because good features come from understanding the data.