Missing Data Disaster

On January 28th, 1986, with the world watching, NASA launched the space shuttle Challenger from the Kennedy Space Center in Florida in freezing temperatures.

Many of us know this part of the Challenger story. But we might not know the data story that was behind it. In fact, the Challenger had also had six critical parts known as O-rings, which “prevent burning rocket fuel from leaking out of booster joints.”¹ There had been seven incidents of distressed O-rings in 23 trial runs leading up to the launch date.

Does this scenario sound familiar?

NASA faced the same quandary the night before as you did in your thought exercise. According to the Rogers Commission report (which was commissioned by President Ronald Reagan after the Challenger accident), a meeting took place the night before launch to discuss the issue.

The managers compared as a function of temperature the flights for which thermal distress of O-rings had been observed—not the frequency of occurrence based on all flights [Figure 4.2].²

“In such a comparison,” the report noted, “there is nothing irregular in the distribution of O-ring ‘distress’ over the spectrum of joint temperatures at launch between 53 degrees Fahrenheit and 75 degrees Fahrenheit.”

NASA went ahead with the launch, based on their review of these failures. But on the day of the launch, the O-rings failed to seal properly in the unusually cold conditions, and the shuttle broke apart 73 seconds into its flight, killing all seven astronauts aboard.

Can you think of any data they missed?

What about the 16 test runs with no failures? Figure 4.3 shows the additional test runs, documented in the Rogers Commission.

Going back to the thought exercise—would you have thought to ask for the missing data? If you had—and perhaps gave it to statisticians to review—you might have seen there was an underlying trend predictive of part failure at lower temperatures. Figure 4.4 shows the trial runs of our autonomous automobile example, including those did not result in critical failures.

Statisticians, engineers, and researchers have studied the Challenger data³ in the decades that followed. And we wanted to present to you a real-world scenario to show that these are the questions data workers must face. One article, published in the Journal of the American Statistical Association (JASA), a leading statistics journal, originally presented the analysis we re-created in Figure 4.4 predicting five out of the six primary O-rings could fail in freezing temperatures. This chart used the data not originally considered the night before launch. The article makes the case that “statistical science could have provided valuable input to the launch decision process.”⁴

Would you have wanted to see this same chart the night before?

The Challenger story offers a chilling scenario of a common phenomenon: that we often look at data that appears to encode the information we need while discarding data we assume wouldn't be relevant. Indeed, we must admit that few situations will ever be as dire as the Challenger scenario. Because so much was at stake, the Challenger scenario serves as an important story that has an immediate and obvious impact.

We don't want to speculate that if only they looked at the full dataset, they would have made the right decision. There is really no way to know this. Other factors were surely at play. Rather, we just want to point out that there are often stories to uncover when we further argue with the data.

And to that extent, the story the Challenger tells is clear. Most businesses don't argue with their data. Instead, they have a culture of acceptance. The effect of this is a slow burn where data projects continue to fail without important questions being asked during the project.

With that, we return to this chapter's goal of teaching you how to argue with the data and what questions to ask.

Who Collected the Data?

When asking who collected the data, we're looking first to establish exactly where the data originated and second, if there are any issues surrounding its origin that would make us ask more follow-up questions.

Many large companies take for granted their data was collected by an internal resource. For instance, a company using workforce data—that is, data based on surveys and associated information of its own employees—might actually be using data owned and collected by a third-party vendor. That last mile of consumption of this data might happen through a portal owned by the company, giving the appearance that the data was collected and is owned by the company, even when it isn't.

In particular, we want you to identify exactly who collected the data. As a Data Head, you must question if external data is reliable and relates to the business problem at hand. Most third-party data is not readily usable in the format it's given to businesses. You, or someone on the data team, will be responsible for transforming the data from a third party into the right structure and format to align with the unique data assets in your company.

How Was the Data Collected?

You should also probe for how data was collected. This question will help uncover if conclusions are being made about the data that aren't allowed. It will also present to you if there are underlying ethical issues with the data collection.

Recall, there are two basic data collection methods: observational and experimental.

Observational data is collected passively. Think website hits, class attendance, and sales numbers. Experimental data is collected under experimental conditions with treatment groups and time-tested precautions to maintain integrity and avoid confounding. Experimental data is the gold standard. Because of the care the experiment provides to ensuring the results are reliable, this data presents an opportunity upon which to derive some causal understanding. For instance, experimental data can help answer the following questions:⁵

• If we give the patient a new drug, will it cure the disease?
• If we discount our product 15%, will it boost next quarter's sales?

Most business data, however, is observational. Observational data should not be used (at least, not exclusively) to derive causal relationships.⁶ Because the data was not collected with specific care toward an experimental design, the usefulness of the data and its underlying results must be presented within this context. Any claims of causality with observational data should be met with skepticism.

Asking how the data was collected will help you uncover whether causality has been assigned when it was not possible to do so. In fact, the incorrect assignment of causality is enough of a problem that we'll return to it several times throughout this book.

It sounds simple enough to use experimental data whenever you can, but to add to the never-ending complexities of working with data, it's not always possible, cost-effective, or even ethical to collect experimental data. For example, if you were assigned to study the impact of “vaping” (smoking electronic cigarettes) on teenagers, you can't randomly assign teens to a treatment and control group and force the treatment group to vape in the name of science. That's not exactly ethical.

As a Data Head, you must work with the data you have while also mediating its ability to drive business decisions. Some companies and departments have the resources to follow up promising observational data with solid experiments. And yet, other business problems do not easily lend themselves to experiments.

Is There Sampling Bias?

Sampling bias happens when the data you have is consistently off or different from the data you care about. Sampling bias is often uncovered indirectly after many decisions have been made on data that is poorly representative of the problem it exists to support. It's only after those decisions continually fail to achieve what is predicted by the data do analysts go back and review if the data was the right data in the first place.

If you want to find out a politician's approval rating and only poll from voters in their political party, you have introduced sampling bias into your data. Good experimental design manages the potential for sampling bias.

In your own work, you may be faced with inherently biased data. Observational data, in particular, is susceptible to bias. The “why was this data collected?” question should uncover why the data exists for your use. Rarely has thought been put into this data to ensure it is free from bias.

You should treat all observational data as inherently biased. You don't need to throw this data out, but you must always present it within the context of its shortcomings.

What Did You Do with Outliers?

Imagine looking at a company's salary data and seeing the number $50,000,000 USD for a new hire in a management role. Would you describe this as an outlier? What would you do with it?

Outliers are defined as data points that differ significantly from other data points. The discovery of outliers should spark a discussion about which data should be rationally excluded from the analysis. Not liking what an extreme value does to an analysis does not automatically mean it should be deleted. For a data point to be removed, have good business justification for removing it.

Arbitrarily picking and choosing which data points are outliers can introduce sampling bias. If outliers are dropped, the original data point and reason for dropping it should be documented and communicated, especially if the results changed substantially.

How Did You Deal with Missing Values?

Missing values are literal holes in a dataset. They represent data points that weren't collected, or may be outliers that were removed (see previous section). Although missing values are a challenge, there are methods to address them. So, it's always worth asking “How did you deal with missing values?”

Suppose you work for a credit card company and you collect data from credit card applicants: name, address, age, employment status, income, monthly housing costs, and number of bank accounts owned. Your job is to predict if these applicants will be late on payments in the next year. Several applicants, however, do not enter their income. So, the system stores it as a blank—a missing value.

Let's go back to the data's origin story. The story starts with applicants applying for a credit card. It's possible the applicant didn't provide their income because they thought they would be denied a credit card if their income was too low. This means the very presence of a missing value might be predictive to whether the application may have a late payment in the future. You don't want to throw out this information!

With this additional insight, a data scientist could create a new categorical feature call “Income Present?” and enter the value 1 if the person entered their income and a 0 if they did not. In this way, you've reencoded missing data into its own categorical variable.

Can the Data Measure What You Want It to Measure?

We often believe we can measure anything and everything. But you should evaluate whether the data provided can truly measure complicated ideas. Consider, for example, the following:

• How would you measure your client's loyalty to your business?
• What data would you use to measure “brand equity” or “reputation?”
• What data can show how much you love your child? Or your pet?

These are hard things to measure. Data, by way of encoding information, enables us to get closer to these answers, but by and large the data we use is truly proxy—a stand in, as it were—for the thing we're trying to measure. The degree to which the proxy reflects reality varies.⁹

To the extent your data indirectly measures something, you should be truthful and honest about it. Measuring complicated concepts like brand equity and reputation require indirect approximations, as these things are truly hard to measure.