Questions to Guide You

To help you embrace the exploratory mindset, we'll walk you through a quick scenario against the backdrop of a popular dataset compiled for educational purposes: the Ames Housing Data.⁴ This is a glimpse into an EDA process.

Although there's no one right path to follow, there are questions you can ask to help guide the team to a meaningful conclusion:

• Can the data answer the question?
• Did you discover any relationships?
• Did you find new opportunities in the data?

Let's set up the scenario and then go through each of the three questions, discuss why they're worth asking, and share challenges you might face.

The Setup

You work for a real estate start-up company and need to drive traffic to your site. But it's hard to pull visitors from real estate tech giants like U.S.-based Zillow.com. Its famous home-value estimation tool, the Zestimate®, brings people (and profits) to Zillow's brand.⁵ To compete, your company needs its own prediction tool. So, you're tasked to build a model that takes a home's information as inputs and produces an estimated sales price as an output.

The boss sends you a dataset to get started. It has 80 columns describing several aspects of hundreds of residential homes that were sold in Ames, Iowa from 2006 to 2011.

Receiving this much data can be overwhelming. However, using the questions outlined previously can help you narrow down how to begin working with the data.

Let's go through them.

Set Expectations and Use Common Sense

You should have a pretty good idea about what information is needed to make an estimate for a home's sales price: size, number of bedrooms, number of bathrooms, the year it was built, etc. These are the popular features home buyers search for on your website. Without them, predicting sales price wouldn't seem reasonable.

You can spot the column names and data types when you open the file. The commonsense features you expect are present, as well as helpful ordinal data (Overall quality of the home, 1–10, 10 being “Very Excellent”), nominal data (Neighborhood), and a host of other features. So far, the data passes an initial sniff test.

Next, you'd likely examine the values the variables take on. Do they cover the scenarios you want to analyze? For example, if you discover the variable “Building Type: Type of Dwelling” only includes single family homes but no apartments, duplexes, or condos, then your model will have limited scope compared to Zillow's. The Zestimate® can predict the sales price of a condo, but if you don't have historical condo data, your company can't reliably predict their sales price.

The lesson: Avoid the fishing expeditions you were warned about in the quote to start this chapter. Make sure the data makes sense for what it's being asked to do.

Do the Values Make Intuitive Sense?

Software will generate a slew of summary statistics for you. Your job is to put the data into context. Check if the summary statistics match your intuitive understanding of the problem. Visualizations are also a key component of EDA—use them to spot anomalies and other weirdness in the data.

Watch Out: Outliers and Missing Values

Every dataset will have anomalies, outliers, and missing values. How you deal with these matters.

For instance, the box plots in Figure 5.2 used a rule-of-thumb to flag several data points as potential outliers. But just because a data graphic classifies certain points as “outliers,” don't turn off critical thinking and automatically delete these points assuming they can't be useful. You'll never catch Zillow removing useful information from its datasets simply because a visualization described these as outliers. Use the context of the data—homes that cost much more than the bulk of other homes is both a known and a common feature of real estate data. Recall the lessons from the previous chapter. You should at least have good business justification to remove outliers. Do you have one here?

And what about missing values? Does a missing value in “Basement Size” mean the house has a basement and the area is unknown? Or does it mean there is no basement and the value should be 0?

If we are diving into the weeds a little, that's our intent. Data workers make hundreds of these tiny decisions during projects. The cumulative effect can be substantial. Left to their own devices—and without the guidance of domain expertise—data workers may continue chipping away at the data, removing difficult and nuanced cases, until the data is too detached from the reality it's trying to capture to be useful. This is why it's important for everyone, including managers, to really understand what their data teams are doing.

Understanding Correlation

Correlation is a measure of how two variables are related. The most common type of correlation used in business is the Pearson correlation coefficient, a statistic between −1 and 1 that measures the linear relationship (think simple straight lines) between pairs of numbers shown on a scatter plot. Correlation can be positive, meaning an increase in one variable is associated with an increase in the other: larger homes sell for more money. Or, correlation can be negative: heavier cars get worse gas mileage. For a visual reference, the correlation of home size and sales price, shown in Figure 5.6, is 0.62. The “tighter” the points around a linear trend, the higher the correlation.⁷

Correlation can help here in two ways. First, finding variables correlated with sales price would help predict it. Second, correlation can help reduce redundancies in your data because two highly correlated variables contain roughly the same information. Imagine two columns in your data: the area of the home in square feet and the area in square meters. These two are perfectly correlated; only one is needed in an analysis.

While most of us have a basic grasp of correlation and report the metric often, it can deceive. Let's review how.

Watch Out: Misinterpreting Correlation

People often forget that correlation is a measure of linear trend, and not all trends are linear.

Suppose, for instance, you're analyzing two neighborhoods in the housing dataset, each with 11 homes. Crunching some statistics reveals that the number of trees on a property is highly correlated with sales price within these neighborhoods. The correlation is a strong 0.8: properties with more trees tend to sell for more money.

But a visual check of the data exposes something unexpected. In Figure 5.7, the data for the neighborhood on the left shows what we'd typically expect to see with a high correlation: a linear trend with data points scattered about. But the plot on the right shows the number of trees is associated with an increase in sales prices only up to a point (11 trees). After that, it trends downward. In the Hilltop community, some properties might have too many trees crowding their lawn.

Full disclosure: the data shown in Figure 5.7 did not come from the Ames dataset we've been exploring but from a popular dataset called Anscombe's Quartet,⁸ four datasets with identical summary statistics but clearly different visualizations. (We are showing just two and adjusted the data to reflect the real estate theme.)

The lesson: Use visualizations to verify noteworthy correlations in your data because the linear trend that correlation can identify may not tell the full story.

Watch Out: Correlation Does Not Imply Causation

Chances are, you've heard the phrase “Correlation does not imply causation” before.¹⁰ But it's worth reiterating because of how much it's ignored and even misunderstood.

When two variables are correlated, even strongly correlated, it does not mean that one is causing the other. Yet people fall into this trap far too often, seeking to build a narrative whenever two variables move together. There are typical silly examples statisticians use to show correlation does not prove causation: Ice cream sales are correlated with shark attacks (both spike in the summer months). Shoe size is correlated with reading ability (both increase over time). But to suggest that reducing ice cream sales will mitigate shark attacks, or that buying bigger shoes helps you read, is clearly a joke. There are other factors at play—outside temperature in the ice cream example, age in the shoe size example—that obviously play a part in the spurious relationships.

But when the correlations aren't built around jokes and the true causal factor isn't clear, the mantra “correlation does not imply causation” is often forgotten.

For example, in real estate data, you find school performance metrics are correlated with home values. Does this mean better schools cause a home's value to increase? Good schools seemingly make a neighborhood more desirable. Or, does the causality go the other direction: higher home prices cause a boost in school performance? Maybe the increase in tax revenue provides more resources to the school. Or does causality go in both directions, creating a feedback loop? Most of the time, we just don't know. There are clearly a multitude of other factors at play, and it'd be rare to have all the answers you need inside your dataset.

It's safer to assume “there is no causality” between two correlated variables unless someone has conducted an experiment proving otherwise. But don't take this to the extreme. Both authors have seen cases in business, university, and media settings where causation is assumed when it shouldn't be. But there are also cases as well where an important association is immediately dismissed as being an assumed causation fallacy. (See the sidebar for an example of causality being dismissed when it should not have been.)