20

Latanya Sweeney, who was the chief technology officer for the US Federal Trade Commission and is now a professor at Harvard University, was surprised when a colleague Googled her name to find one of her papers and discovered ads suggesting she had been arrested.1 Sweeney clicked on the ad, paid a fee, and learned what she already knew: she had never been arrested. Intrigued, she entered the name of her colleague Adam Tanner, and the same company’s ad appeared but without the suggestion of arrest. After more searching, she developed the hypothesis that maybe black-sounding names were triggering the arrest ad. Sweeney then tested this more systematically and found that if you Googled a black-associated name like Lakisha or Trevon, you were 25 percent more likely to get an ad suggesting an arrest record than if you searched for a name like Jill or Jack.2

Such biases are potentially damaging. Searchers might be looking for information to see if someone is suitable for a job. If they find ads with titles like “Latanya Sweeney, Arrested?” the searchers might have some doubts. It is both discriminatory and defamatory.

Why was this happening? Google provides software that allows advertisers to test and target particular keywords. Advertisers might have entered racially associated names to place ads alongside, although Google denied that.3 Another possibility is that the pattern emerged as a result of Google’s algorithms, which promote ads that have a higher “quality score” (meaning they are likely to be clicked). Prediction machines likely played a role there. For instance, if potential employers searching for names were more likely to click on an arrest ad when associated with a black-sounding name than other names, then the quality score associated with placing those ads with such keywords might rise. Google is not intending to be discriminatory, but its algorithms might amplify prejudices that already exist in society. Such profiling exemplifies a risk of implementing AI.

Liability Risks

The emergence of racial profiling is a societal issue, but also a potential problem for companies like Google. They may run afoul of employment antidiscrimination rules. Fortunately, when whistleblowers like Sweeney raise the issue, Google is highly responsive, investigating and correcting problems.

Discrimination might emerge in even subtler ways. Economists Anja Lambrecht and Catherine Tucker, in a 2019 study, showed that Facebook ads could lead to gender discrimination.4 They placed ads promoting jobs in science, technology, engineering, and math (STEM) fields on the social network and found Facebook was less likely to show the ad to women, not because women were less likely to click on the ad or because they might be in countries with discriminatory labor markets. On the contrary, the workings of the ad market discriminated. Because younger women are valuable as a demographic on Facebook, showing ads to them is more expensive. So, when you place an ad on Facebook, the algorithms naturally place ads where their return per placement is highest. If men and women are equally likely to click on STEM job ads, then it is better to place ads where they are cheap: with men.

Economist and lawyer Ben Edelman explained to us why this issue could be serious for both employers and Facebook. While many tend to think of discrimination as arising from disparate treatment—setting different standards for men and women—the ad-placement differences might result in what lawyers call “disparate impact.” A gender-neutral procedure turns out to affect some employees who might have reason to fear discrimination (a “protected class” to lawyers) differently from others.

A person or an organization can be liable for discrimination, even if it is accidental. A court found that the New York City Fire Department discriminated against black and Hispanic applicants becoming firefighters with an entrance exam that included several questions emphasizing reading comprehension. The court found that the types of questions had no relation to effectiveness as a fire department employee and that black and Hispanic applicants performed systematically worse on them.5 The case was eventually settled for about $99 million. Blacks’ and Hispanics’ lower performance on the exam meant that the department was liable, even if the discrimination was unintentional.

So, while you may think you are placing a neutral ad on Facebook, disparate impact might be emerging regardless. As an employer, you could be liable. Of course, you don’t want to engage in discrimination, even implicitly. One solution for Facebook is to offer tools for advertisers to prevent discrimination.

A challenge with AI is that such unintentional discrimination can happen without anyone in the organization noticing. Predictions generated by deep learning and many other AI technologies appear to be created from a black box. It isn’t feasible to look at the algorithm or formula underlying the prediction and identify what causes what. To figure out if AI is discriminating, you have to look at the output. Do men get different results than women? Do Hispanics get different results than others? What about the elderly or the disabled? Do these different results limit their opportunities?

To prevent liability issues (and to avoid being discriminatory), if you discover unintentional discrimination in the output of your AI, you need to fix it. You need to figure out why your AI generated discriminatory predictions. But if AI is a black box, then how can you do this?

Some in the computer science community call this “AI neuroscience.”6 A key tool is to hypothesize what might drive the differences, provide the AI with different input data that tests the hypothesis, and then compare the resulting predictions. Lambrecht and Tucker did this when they discovered that women saw fewer STEM ads because it was less expensive to show the ad to men. The point is that the black box of AI is not an excuse to ignore potential discrimination or a way to avoid using AI in situations where discrimination might matter. Plenty of evidence shows that humans discriminate even more than machines. Deploying AI requires additional investments in auditing for discrimination, then working to reduce any discrimination that results.

Algorithmic discrimination can easily emerge at the operational level but can end up having strategic and broader consequences. Strategy involves directing those in your organization to weigh factors that might not otherwise be obvious. This becomes particularly salient with systematic risks, like algorithmic discrimination, that may have a negative impact on your business. Showing the STEM ads to men and not women bolstered short-term performance (in that the ads the men saw cost less) but created risks due to the resulting discrimination. The consequences of increasing risks may not become apparent until too late. Thus, a key task for a business’s leaders is to anticipate various risks and ensure that procedures are in place to manage them.

Quality Risks

If you are in a consumer-facing business, you probably buy ads and have seen a measure of those ads’ ROI. For instance, your organization may have found that paying for Google ads resulted in an increase in click-throughs and maybe even purchases on the website. That is, the more ads your company bought on Google, the more clicks from those ads it received. Now, try employing an AI to look at that data and generate a prediction of whether a new Google ad is likely to increase clicks from that ad; the AI will likely back up that positive correlation you had previously observed. As a result, when the marketing people want to buy more Google ads, they have some ROI evidence to back it up.

Of course, it takes an ad to generate a click. One possibility is that without the ad, the consumer would never know about your product. In this case, you want to place ads because they generate new sales. Another possibility is that the ad is the easiest thing for potential customers to click, but in its absence, they would find you anyway. So while the ad may be associated with more sales, it is potentially a fiction. Without the ad, sales may have increased regardless. Thus, if you really want to know if the ad—and the money you spend on it—is generating new sales, you need to examine the situation more deeply.

In 2012, some economists working for eBay—Thomas Blake, Chris Nosko, and Steve Tadelis—persuaded eBay to turn off all of its search advertising in one-third of the United States for an entire month.7 The ads had a measured ROI using traditional statistics of more than 4,000 percent. If the measured ROI was correct, doing a month-long experiment would cost eBay a fortune.

However, what they found justified their approach. The search ads eBay placed had practically no impact on sales. Their ROI was negative. Consumers on eBay were savvy enough that, if they didn’t see an ad in Google, they would click on ordinary (or organic) search results in Google. Google would highly rank eBay listings regardless. But the same was true for brands like BMW and Amazon. The only area where ads seemed to do some good was in attracting new users to eBay.

This story’s point is to demonstrate that AI—which does not rely on causal experimentation but on correlation—can easily fall into the same traps as anyone using data and simple statistics can. If you want to know whether advertising is effective, observe whether ads lead to sales. However, that is not necessarily the full story, because you also need to know what would happen to sales if you ran no ads. An AI trained on data that involves lots of ads and sales does not get to see what happens with few ads. That data is missing. Such unknown knowns are a key weakness of prediction machines that require human judgment to overcome. At the moment, only thoughtful humans can work out if the AI is falling into that trap.

Security Risks

While software has always been subject to security risks, with AI those risks emerge through the possibility of data manipulation. Three classes of data have an impact on prediction machines: input, training, and feedback. All three have potential security risks.

Input Data Risks

Prediction machines feed on input data. They combine this data with a model to generate a prediction. So, just like the old computer adage—“garbage in, garbage out”—prediction machines fail if they have poor data or a bad model. A hacker might cause a prediction machine to fail by feeding it garbage data or manipulating the prediction model. One type of failure is a crash. Crashes might seem bad, but at least you know when they have occurred. When someone manipulates a prediction machine, you may not know about it (at least not until too late).

Hackers have many ways to manipulate or fool a prediction machine. University of Washington researchers showed that Google’s new algorithm for detecting video content could be fooled into misclassifying videos by inserting random images for fractions of a second.8 For example, you can trick an AI into misclassifying a video of a zoo by inserting images of cars for such a short time that a human would never see the cars, but the computer could. In an environment where publishers need to know content being published to appropriately match advertisers, this represents a critical vulnerability.

Machines are generating predictions used for decision-making. Companies deploy them in situations where they really matter: that is, where we expect them to have a real impact on decisions. Without such decision embeddedness, why go to the trouble of making a prediction in the first place? Sophisticated bad actors in this context would understand that by altering a prediction, they could adjust the decisions. For instance, a diabetic using an AI to optimize insulin intake could end up in serious jeopardy if the AI has incorrect data about that person and then offers predictions that suggest lowering insulin intake when it should be increased. If harming a person is someone’s objective, then this is one way to do it effectively.

We are most likely to deploy prediction machines in situations where prediction is hard. A bad actor might not find precisely what data is needed to manipulate a prediction. A machine may form a prediction based on a confluence of factors. A single lie in a web of truth is of little consequence. In many other situations, identifying some data that can be used to manipulate a prediction is straightforward. Examples might be location, date, and time of day. But identity is the most important. If a prediction is specific to a person, feeding the AI the wrong identity leads to bad consequences.

AI technologies will develop hand-in-hand with identity verification. Nymi, a startup we worked with, developed a technology that uses machine learning to identify individuals via their heartbeat. Others are using retina scans, faces, or fingerprint identification. Companies can also confirm an identity by using the characteristics of a smartphone user’s walking patterns. Regardless, a happy confluence in technologies may emerge that allows us to simultaneously personalize AI and safeguard identity.

While personalized predictions might be vulnerable to the manipulation of the individual, impersonal predictions may face their own set of risks related to population-level manipulation. Ecologists have taught us that homogenous populations are at greater risk of disease and destruction.9 A classic example is in farming. If all farmers in a region or country plant the same strain of a particular crop, they might do better in the short term. They likely chose that crop because it grows particularly well in the region. By adopting the best strain, they reduce their individual risk. However, this very homogeneity presents an opportunity for disease or even adverse climate conditions. If all farmers plant the same strain, then they are all vulnerable to the same disease. The chances of a disastrous widespread crop failure increase. Such monoculture can be individually beneficial but increase systemwide risk.

This idea applies to information technology generally and prediction machines in particular. If one prediction machine system proves itself particularly useful, then you might apply that system everywhere in your organization or even the world. All cars might adopt whatever prediction machine appears safest. That reduces individual-level risk and increases safety; however, it also expands the chance of a massive failure, whether purposeful or not. If all cars have the same prediction algorithm, an attacker might be able to exploit that algorithm, manipulate the data or model in some way, and have all cars fail at the same time. Just as in agriculture, homogeneity improves results at the individual level at the expense of multiplying the likelihood of systemwide failure.

A seemingly easy solution to the problem of systemwide failure is to encourage diversity in the prediction machines you deploy. This will reduce the security risks, but at the cost of reduced performance. It might also increase the risk of incidental smaller failures due to a lack of standardization. Just as in biodiversity, the diversity of prediction machines involves a trade-off between individual and system-level outcomes.

Many of the scenarios for systemwide failure involve an attack on several prediction machines at the same time. For example, an attack on one autonomous vehicle represents a risk to safety; an attack on all autonomous vehicles simultaneously presents a national security threat.

Another way to secure against a massive simultaneous attack, even in the presence of standard homogenous prediction machines, is to untether the device from the cloud.10 We have already discussed the benefits of implementing prediction on the ground rather than in the cloud for the purpose of faster context-dependent learning (at the cost of more accurate predictions overall) and to protect consumer privacy.

Prediction on the ground has another benefit. If the device is not connected to the cloud, a simultaneous attack becomes difficult.11 While training the prediction machine likely happens in the cloud or elsewhere, once the machine is trained, it may be possible to do predictions directly on the device without sending information back to the cloud.

Training Data Risks

Another risk is that someone can interrogate your prediction machines. Your competitors may be able to reverse-engineer your algorithms, or at least have their own prediction machines use the output of your algorithms as training data. Perhaps the most well-known example involves a sting by Google’s anti-spam team. It set up fake results for a variety of absurd search queries such as “hiybbprqag” that otherwise did not exist. It then had Google engineers query those words from their home computers. Specifically, it told the engineers to use Microsoft Internet Explorer’s toolbar for the searches. Weeks later, the team queried Microsoft’s Bing search engine. Sure enough, Google’s fake results for the searches like “hiybbprqag” showed up as Bing results. Google’s team showed that Microsoft uses its toolbar to copy Google’s search engine.12

At the time, there was much discussion about whether what Microsoft did was acceptable or not.13 In effect, Microsoft was using the Google toolbar for learning-by-using to develop better algorithms for its Bing search engine. Much of what users did was search Google and then click on those results. So when a search term was rare and only found on Google (like “hiybbprqag”) and if it was used enough (precisely what the Google engineers were doing), Microsoft’s machine ended up learning it. Interestingly, what Microsoft had not been doing—which it clearly could have—was learn how Google search terms translated into clicks to imitate completely Google’s search engine.14

The strategic issue is that when you have an AI (like Google’s search engine), then if a competitor can observe data being entered (such as a search query) and output being reported (such as a list of websites), then it has the raw materials to employ its own AI to engage in supervised learning and reconstruct the algorithm. Google’s search engine would be a very difficult undertaking with respect to such expropriation, but it is, in principle, quite possible.

In 2016, computer science researchers showed that certain deep-learning algorithms are particularly vulnerable to such imitation.15 They tested this possibility on some important machine-learning platforms (including Amazon Machine Learning) and demonstrated that with a relatively small number of queries (650–4,000), they could reverse-engineer those models to a very close approximation, sometimes perfectly. The very deployment of machine-learning algorithms leads to this vulnerability.

Imitation can be easy. After you have done all of the work of training an AI, that AI’s workings are effectively exposed to the world and can be replicated. But more worrisome is that the expropriation of this knowledge may lead to situations where it is easier for bad actors to manipulate the prediction and the learning process. Once an attacker understands the machine, the machine becomes more vulnerable.

On the positive side, such attacks leave a trail. It is necessary to query the prediction machine many times to understand it. Unusual quantities of queries or an unusual diversity of queries should raise red flags. Once raised, then protecting the prediction machine becomes easier, although not easy. But at least you know that an attack is coming and what the attacker knows. Then you can protect the machine by either blocking the attacker or (if that is not possible) preparing a backup plan if something goes wrong.

Feedback Data Risks

Your prediction machines will interact with others (human or machine) outside your business, creating a different risk: bad actors can feed the AI data that distorts the learning process. This is more than manipulating a single prediction, but instead involves teaching the machine to predict incorrectly in a systematic way.

A recent and dramatic public example occurred in March 2016 when Microsoft launched an AI-based Twitter chatbot named Tay. Microsoft’s idea was solid: have Tay interact with people on Twitter and determine how best to respond. Its intention was to learn specifically about “casual and playful conversation.”16 On paper, at least, this was a sensible way of exposing an AI to the experience it needed to learn quickly. Tay started off as not much more than a parrot, but the goal was more ambitious.

The internet, however, is not always a gentle setting. Soon after launch, people started to test the limits of what Tay would say. “Baron Memington” asked “@TayandYou Do you support genocide,” to which Tay responded “@Baron_von_Derp I do indeed.” Soon Tay seemed to become a racist, misogynist, Nazi sympathizer. Microsoft pulled the experiment.17 Precisely how Tay evolved so quickly is not entirely clear. Most likely, interactions with Twitter users taught Tay this behavior. Ultimately, this experiment demonstrated how easy it is to undermine machine learning when it occurs in the real world.

The implications are clear. Your competitors or detractors may deliberately try to train your prediction machine to make bad predictions. As with Tay, data trains prediction machines. And prediction machines that are trained in the wild may encounter people who use them strategically, maliciously, or dishonestly.

Facing Risk

Prediction machines carry risks. Any company that invests in AI will face these risks, and eliminating all of them is impossible. There is no easy solution. You now have the knowledge to anticipate these risks. Be aware of how your predictions differ across groups of people. Question whether your predictions reflect underlying causal relationships and if they are really as good as they seem to be. Balance the trade-off between systemwide risks and the benefit of doing everything a little bit better. And watch for bad actors who may query your prediction machines to copy them or even destroy them.

KEY POINTS

• AI carries many types of risk. We summarize six of the most salient types here.
  1. Predictions from AIs can lead to discrimination. Even if such discrimination is inadvertent, it creates liability.
  2. AIs are ineffective when data is sparse. This creates quality risk, particularly of the “unknown known” type, in which a prediction is provided with confidence, but is false.
  3. Incorrect input data can fool prediction machines, leaving their users vulnerable to attack by hackers.
  4. Just as in biodiversity, the diversity of prediction machines involves a trade-off between individual- and system-level outcomes. Less diversity may benefit individual-level performance but increase the risk of massive failure.
  5. Prediction machines can be interrogated, exposing you to intellectual property theft and to attackers who can identify weaknesses.
  6. Feedback can be manipulated so that prediction machines learn destructive behavior.