How does prediction work

Think about how we go about doing predictions on a day-to-day basis. One very common situation that almost everyone can relate to is deciding at what time to leave to make it to work on time. The first time we did it we may have asked others and taken up their recommendation.

Alternatively, we probably preferred to play it safe and left really early, most likely overestimating the time it would take us to make it to the office. After a while, after driving or taking the train over and over, we learned to make more reliable predictions. Here, as in any other prediction problem, we learn from our environment and adjust our predictions accordingly.

It shouldn’t come as a suprise, then, that machine-created predictions apply the same principle: we feed data into algorithms that are designed to provide accurate predictions when some conditions take place.

Of course, we need to precisely define what “accurate” means, and if you’re interested, in the Appendix I provide a more formal introduction to ML. But for now it will suffice to say that the vast majority of learning algorithms we use at the enterpise work as follows:

1. Start creating a model of how the world works — or more precisely, how the outcome we want to predict works.

2. Feed data that relates closely to that model and make a prediction using the model.

3. Compare the prediction with the actual outcome and define a criterion to identify how good the prediction is.

4. Using some sensible way to update our predictions, calibrate the model until we feel comfortable with the prediction accuracy.

Let’s put this basic learning algorithm in practice using our daily commute example:

1. The time it takes me to get to my destination depends on the distance and the speed. Speed itself depends on the amount of traffic and, we conjecture, follows a U shape: it is high when there is no traffic, decreases to a minimum at peak hours and increases afterwards. It follows that commute time has an inverted U shape on the amount of traffic, reaching a maximum during rush hour.

2. Try leaving at 6:30 am and measure the time it takes to get to the office. Repeat at 6:45 am, 6:57 am, 7:15 am and so on. Our input data will be the departing time and the output we wish to predict is the commute time.

3. Make an initial guess of our inverted U-shape model to predict commute time if I leave tomorrow at 6:50. Leave at 6:50 tomorrow, measure the time, and compare with the actual time.

4. Recalibrate the U-shape model and repeat.

What I just described is called a “supervised” learning algorithm. In steps 3 and 4, our experience helps to supervise the calibration process since we are able to measure precisely how good or bad our predictions were.

This process is replicated, at scale, using computing power and smart updating rules when we use supervised machine learning algorithms. These are at the core of the modern data scientist toolkit.

Why is everyone talking about AI today: Deep Learning

AI has become a buzz word thanks to the predictive power of deep learning algorithms. This is a shortcut for deep neural network learning algorithms. In the Appendix I describe how these work so I won’t deviate our focus right now by delving into the technical details.

The important thing to notice is that these have become very powerful in tackling problems that previously (less than a decade ago) were very hard to solve using machines. And by “very hard” I mean that humans significantly outperformed the best existing algorithms. In many domains this is no longer the case and the best deep learning algorithms have reached super human predictive power.

Two examples where deep learning excels are in the domains of image recognition and language processing. It is interesting to discuss how prediction comes about in tackling these problems as this will reinforce the idea that current AI are just predictive algorithms.

Better predictions may produce better decisions

At a first level, predictive technologies will improve the accuracy of our decisions, as most relevant decisions are taken under uncertainty where we, consciously or not, solve a predictive problem. For instance, when you decided to buy this book you must have thought that it was worth your money and your time. Your decision (buying) was fed with a prediction (quality), so if we make better predictions we should also make better decisions.

How you approached and solved this problem is unclear, but most likely you did not consciously pose it as predictive question. Perhaps you used a set of heuristics or shortcuts to decide if it was worth it for you: maybe O’Reilly’s reputation as a high-quality publisher gave you some confidence, you read some positive reviews, or you found the title catchy. These are all shortcuts we constantly use to solve potentially low-risk predictive questions. For higher-risk problems we may use some more explicit conscious reasoning.

Note that in the title to this section I have emhasized the verb “may”. It is not straightforward that improving the accuracy of our predictions generate better decisions. You must act on this evidence, that is, your decision must depend on these predictions. Many companies hire armies of data scientists that create no value whatsoever because of this problem.

Data translators: identifying business cases for prediction

It seems safe to anticipate, then, that this ability to identify and prioritize different predictive opportunities will soon become of very high value to companies. Taking this skill one step ahead, if you can also translate these business problems to predictive problems that can be solved by data scientists your value for the company will be much higher. One study from the McKinsey Global Institute estimates the demand for data translators in the United States alone to be between 2 and 4 million by 2026.²

Moreover, companies that can adapt their business models early enough, in order to take advantage of these advances should be able to capture a significant share of the market. We are already seeing this with many startups that are built upon AI capabilities, and the speed of adoption of other more consolidated companies will depend on the competitive challenges they face from these newcomers and also on their executives’ and higher-management vision into the future. The banking and telco industries are two examples where disruption is forcing the traditional players to embark on widespread transformations.

Job automation and demand for complementary skills

At a second level, the process of task automation is on its way and will affect the ways companies operate. Take Robot Process Automating or RPA, where highly repeatable and stable actions are automated with the help of a computer program. RPA’s success comes from automation in low-uncertainty environments. More complex actions are also being automated with the help of the current wave of prediction technologies and this trend is only going to increase. The 2018 World Economic Forum´s Future of Jobs Report shows that 50% of interviewed companies expect automation to reduce their workforce by 2022.³ A study by the UK’s Office for National Statistics estimates that 1.5 million of current jobs (7.4%) are at risk of automation.⁴ Another worldwide analysis puts the figure between 75 and 375 million by 2030.⁵

As machines continue to substitute humans in performing certain tasks, demand for complementary skills that are not easily automated will most likely increase. The above-mentioned Future of Jobs Report ranks number one the skill of “analytical thinking and innovation” both at the time of the study and up to 2022, their most futuristic forecast. Machines are very good at solving prediction problems once humans have translated the problem, so analytic translation is one such complementary skill that will be on the rise.

The 3 V’s

The first pillar involved the now well-known three Vs: volume, variety and velocity. The internet transformation had provided companies with ever-increasing volumes of data. One 2018 estimate claims that in the previous two years, 90% of the data created in the history of human kind had been generated, and many such calculations abound.⁷ Technology had to adapt if we wanted to analyze this apparently unlimited supply of information: we not only had to store and process larger amounts of data, but also needed to deal with new unstructured types of data such as text, images, videos and recordings that were not easily stored or processed with the data infrastructure available at the time.

After the innovations were made, consultants and vendors had to invent new ways to market the new technologies. For structured data we had the Enterprise Data Warehouse, so they created a new catchy term. The Data Lake was thus born with the promise of providing flexibility and computational power to store and analyze big data.

Flexibility came in two flavors: thanks to “linear scalability”, if twice the work needed to be done, we would just have to install twice the computing power to meet the same deadlines. Similarly, for a given task, we could cut the current time by half by doubling the amount of infrastructure. Computing power could be easily added by way of commodity hardware, efficiently operated by ever-more-clever open-source software readily available for us to use. But the data lake also allowed for quick access to the larger variety of data sources.

Once we tackled the volume and variety problems, velocity was the next frontier, and our objective had to be the reduction of time-to-action and time-to-decision. We were now able to store and process large amounts of very diverse data in real-time or near real-time if necessary. The three Vs were readily achievable.

Data-driven culture

I have already devoted some time to the second pillar in the big data revolution, predictive technologies, so let’s talk about the third pillar: culture. A company can buy the best technology to process large amounts of data, can hire the best data scientists and data engineers to create and automate machine learning data flows or pipelines, but if it doesn’t act on this evidence little can be accomplished. A data-driven organization is one that acts on data, or put differently, one that operates their business using evidence-based decision-making. I prefer the second definition since it takes the emphasis from data and puts it on evidence and the scientific method. Core cultural values are experimentation, curiosity, empowerment, data democratization and agility.

From pillars to recipes for success

The three Vs were physical descriptions of the data that we could now process; but they were not recipes to create value in the new data economy. The other two pillars were also enablers, but we needed futher guidance. Consultants and vendors alike provided us then with maturity models, roadmaps to be followed to ascend to paradise. One such model is depicted in Figure 1-1, which I will explain now.

Figure 1-1. A possible data maturity model showing a hierarchy of value creation

A tale of unrealized promises

The three pillars, along with data maturity models similar to the one just described, have been part of the conversation during the last 15 years or so. This conversation has been largely directed by consulting firms and given the size of the promised economic gains, most companies have followed suit.

Have the promises of unlimited data-driven value been realized? Is data the new oil, as The Economist famously claimed a couple of years ago?⁸ Let’s start with what the market thinks: Figure 1-2 shows the evolution of the top 10 companies by market capitalization, that is, by taking the total outstanding stock and valuing them at their market prices. With the probable exception of Berkshire Hathaway — Warren Buffett’s conglomerate — , Johnson & Johnson (J&J) and JP Morgan, all of the remaining companies are in the technology sector and all have embraced the data and AI revolution.⁹ This does not prove that big data is the cause of this unprecedented wealth, but at the minimum it shows that the markets’ top performers have fully adopted the new credo.

Figure 1-2. Evolution of market capitalization top-10 ranking

One 2011 report by the McKinsey Global Institute estimated sector-specific additional productivity growth rates between 0.5 and 1% from the use of Big Data.¹⁰ Their Big Data Value Potential Index positions finance, government and information sectors top, followed by wholesale trade and real estate. As the name suggests, these are estimates of potential and not realized gains. Another study estimates that becoming more data-driven increases a firm’s productivity by 5-6%.¹¹

But after almost 15 years since the data revolution got kick-started most companies have not realized the promise of unprecedented value creation. According to one recent survey with more than 60 executives from leading firms, even though 92% report speeding up their AI and data investments, only 31% of them self-identify as data-driven — falling from 37% two years earlier — , and more than half admit that data is not treated as an asset in their enterprises. This results in 37% reporting that they have not yet found measurable results from their AI and data initiatives, and 77% that their adoption is still a challenge.¹²

The study also cites some factors that explains this lack of results: lack of organizational alignment, cultural resistance, misunderstanding the role that data plays as an asset for the company, faulty executive leadership and technological barriers. Similar results abound in a growing literature that tries to explain this paradoxical state: if everyone agrees that the data and AI revolutions can potentially generate considerable value, why can’t companies reach this potential? The promised fortunes are like the never-reachable tortoise in Zeno’s paradox, and we, of course, the frustrated Achilles.

The central tenet of this book is that one often-overlooked cause for this lack of results come from the general lack of analytical skills within the organizations. And the modern AI-driven economy needs a better definition of what thinking analytically means.