Exploratory data analysis (EDA) is the structured first look at a dataset: understanding what each column means, checking data quality, summarising single variables, examining relationships between variables and investigating outliers, before modelling or reporting. Good EDA catches data problems early, suggests features and hypotheses, and prevents confident conclusions built on broken data.
Why does EDA matter so much?
Real data is messy: missing values, duplicates, inconsistent spellings, impossible values, changed definitions over time, and leakage hiding in innocent-looking columns. A model trained on such data will happily learn its mistakes. EDA is where you discover that "0" means "unknown" in one column, that one city appears under three spellings, or that a column was only populated after March. Finding these before modelling saves weeks and protects your credibility.
Step 1: what context do you need first?
Before touching code, ask: where did the data come from, how was it collected, what does one row represent, what period does it cover, and what decision will the analysis support? A dataset of "students" might have one row per student, per enrolment or per month. Getting this wrong invalidates everything after it. Read any data dictionary, and talk to whoever produced the data if you can.
Step 2: how do you inspect structure?
Load the data and check shape, column names, data types and a sample of rows. In Pandas: df.shape, df.info(), df.head(), df.sample(10). Look for numbers stored as text, dates stored as strings, and IDs read as numbers (which can lose leading zeros). Fix types before going further. See the Pandas tutorial for the commands.
Step 3: what quality checks should you run?
| Check | How | What you might find |
|---|---|---|
| Missing values | df.isna().mean() per column | A column 80% empty; missingness concentrated in one source |
| Duplicates | df.duplicated().sum(), duplicate IDs | Repeated orders from a retry bug |
| Impossible values | min/max, rule checks | Age 0 or 200; negative prices; future dates |
| Inconsistent categories | value_counts() | "Kolkata", "kolkata", "Calcutta" |
| Placeholder values | Frequent suspicious values | "9999", "N/A", "0" meaning unknown |
| Time coverage | Counts per day or month | Gaps, sudden jumps when a system changed |
Step 4: how do you analyse single variables?
For each important numeric column: summary statistics (mean, median, standard deviation, percentiles) and a histogram or box plot to see shape, skew and outliers. For each categorical column: counts and percentages, the number of distinct values, and how many rare categories there are. For the target variable in a modelling project: its distribution and, for classification, the class balance, which determines which metrics make sense.
Illustrative numbers. A chart like this, made in one line of Pandas, is often the first slide of an EDA summary.
Step 5: how do you analyse relationships?
- Numeric vs numeric: scatter plots and correlation (Pearson for linear, Spearman for monotonic relationships). Watch for non-linear patterns that correlation misses.
- Categorical vs numeric: grouped box plots or averages by group, such as completion rate by city tier.
- Categorical vs categorical: cross-tabulations with percentages.
- Feature vs target: the most important view in modelling, which suggests useful features and reveals suspicious ones (a feature that predicts the target almost perfectly is often leakage).
- Over time: plot key metrics by week or month to spot trends, seasonality and breaks.
Step 6: what do you do with outliers and missing data?
Investigate before acting. An outlier may be a data-entry error (fix or remove), a genuine rare case (keep, perhaps cap for some models) or the most important signal in the data (fraud!). For missing values, ask why they're missing: randomly, or for a reason related to the outcome? Options include dropping rows or columns, filling with a median or "Unknown", model-based imputation, and adding a "was missing" indicator, which often carries useful information.
Step 7: how do you record and share findings?
End every EDA with a short written summary: what the data contains, the quality issues found and how you handled them, the main patterns, open questions, and the hypotheses or features to test next. Keep cleaning steps in code (not manual edits) so they can be rerun on new data. This document becomes the foundation for modelling, feature engineering and stakeholder conversations.
What does an EDA checklist look like?
- Know what one row represents and the time period covered.
- Correct data types.
- Missing values quantified and explained.
- Duplicates and impossible values checked.
- Categories standardised.
- Distributions of key variables and the target viewed.
- Relationships with the target explored; leakage suspects flagged.
- Trends over time checked.
- Findings and next steps written down.
Are automated EDA tools worth using?
Profiling libraries generate a full report of distributions, correlations, missing values and warnings in one command. They're excellent for a quick first overview of a new dataset and for spotting columns that need attention. They don't replace thinking: they can't tell you what a column means, whether a pattern makes business sense, or that a "perfect" predictor is leakage. Use them to direct your attention, then investigate the interesting parts yourself.
What mistakes are common in EDA?
- Skipping context and misreading what a row means.
- Treating correlation as causation.
- Deleting outliers without investigating them.
- Looking only at averages and never at distributions.
- Making manual fixes in a spreadsheet that can't be reproduced.
- Exploring the test set, which leaks information into modelling decisions.
What does EDA look like on a real example?
Suppose you receive a year of course-enrolment data to understand drop-outs. Context first: each row is one enrolment, and "dropped" means no activity for 30 days. Structure: the signup date is text, so you convert it. Quality: 3% of rows are duplicates from a payment retry, and the "city" column has 40 spellings of 12 cities. Single variables: most learners are aged 18–30, and the drop-out rate is about a quarter. Relationships: drop-out is much lower for learners who attend the first live class, and higher for those who sign up late at night (a hypothesis worth testing, not a conclusion). Time: enrolments spike before batch start dates. Your summary lists the fixes, these patterns and three hypotheses for a model or an experiment. That's a complete, useful EDA in a few hours.
How does EDA differ for text, images and time series?
For text, look at length distributions, language mix, the most frequent words and phrases, duplicates, and examples from each label. For images, view random samples per class, check resolutions, brightness and orientation, and look for near-duplicates and mislabelled images. For time series, plot the raw series, rolling averages and seasonal patterns, check for gaps and sudden level changes, and note holidays or events that explain spikes. The goal is the same in every case: understand what the model will learn from before it learns it. Ten minutes looking at raw examples routinely reveals problems that no summary statistic would.
Finally, rerun your EDA whenever new data arrives: patterns and problems change over time, and last quarter's checks may miss this quarter's issues.
How should you practise?
Pick a messy public dataset, from government open-data portals, for example, and run through the checklist, writing a one-page summary. EDA with NumPy and Pandas opens the data science phase of Program Zero's data science course; continue with data visualisation in Python.