The Importance of Exploratory Data Analysis

Most data science projects fail.

And when I say most, I mean 90% of them. More concerning is that only 20% of analytics insights deliver business outcomes. Even in the age of AI, when the veil of knowledge has been removed by a string of commmands that can give you answers that could only be dreamed of a few years ago, projects are falling to the wayside.

There are plenty of reasons to why data science projects fail. But in this post, I want to focus on the main one, and to me, the only reason—EDA.

Across the many projects I’ve worked on, I’ve noticed one common trend: teams are eager to accelerate past exploratory data analysis (EDA), overlooking the fact that EDA is the body and heart of data science; mathematics is the head.

EDA is a tedious process, and when done properly, a DS would spend no less than 85% of the time in this phase. It is the most critical phase for the right output.

What is Exploratory Data Analysis

EDA is how we scrutinize and probe the data to understand their primary features. This is the time where we get familiar with the data and understand what it can offer.

In certain cases, EDA allows us to validade or refute hypothesis, and scrutinize underlying assumptions.

The idea of EDA was introduced by mathematician John Tukey in the 70s. The method acts like a detective work— allowing data to reveal its underlying structure rather than trying to fit it into a preconceived hypothesis. Fifty years later, the idea works still works.

The core objectives are:

Understand the structure: learn the size of the data, data types, and basic descriptive statistics.
Detect errors and gaps: identify missing values, duplicate entries, corrupted data points, etc.
Find outliers: Finding anomalies that might skew the machine learning output.
Uncover relationships: See how different variables interact or correlate.

A number of techniques are available to get the job done. Mostly a myriad of visualization tools. One of the most common is matplotlib, which allows us to visualize things like standard diviation, anomoalies, unearth patterns, and so on.

Why is EDA Important

When you are starting to work on a project and have never seen the data, you have no idea of what is in it. The main purpose of EDA is so that you can look into the data and make solid assumptions and identify obvious errors.

More importantly, EDA will help determine if the business problem or question can be answered with the data in hand. It is also the most important step before applying any kind of statistical analysis or modeling, including machine learning.

EDA Tools

Let’s talk about some of the tools for EDA.

Univariate visualizations. Provides you with summary statistics of each field in the raw data.
Mutivarita visualizations. Maps out the interactions between different fields in the data
Clustering and dimension reduction. Create graphics with high-dimensional data containing many variables.
Linear regression. Simple model that helps predict outcomes.

While EDA can be conducted using several different tools, the most common are the programming languages: R and Python.