A Big Bag of Words

The most basic approach to convert text into numbers is by creating a “bag of words” model. In a bag of words, a sentence of text is jumbled together into a “bag” where word order and grammar are ignored. What you read as: “This sentence is a big bag of words” is converted into a set, called a document, where each word is an identifier, and the count of the word is a feature. Order does not matter, so we'll sort the bag alphabetically by identifier:

{a: 1, bag: 1, big: 1, is: 1, of: 1, sentence: 1, this: 1, words: 1}.

Each identifier is referred to as a token. The entire set of tokens from all documents is called a dictionary.

Of course, your text data will contain more than one document, so this bag of words can get exceedingly large. Every unique word and spelling would become a new token. Here's how this would look as a table, where each row contains a sentence (or customer comment, product review, etc.).

For the raw text:

• This sentence is a big bag of words.
• This is a big bag of groceries.
• Your sentence is two years.

The bag-of-words representation would look like that in Table 11.1, where the data points represent the count of each word in the sentence.

Looking at Table 11.1, called a document-term matrix (one document per row, one term per column), you'll notice how easy it would be to perform some basic text analytics: calculating summary statistics of each word (“is” is the most popular word) and finding out which sentence contains the most tokens (first sentence). While not interesting in this example, this is how basic summary statistics of documents can be calculated.

We suspect you've also noticed some drawbacks in Table 11.1 (and hopefully word clouds!). As more documents are added, the number of columns in the table would become exceedingly wide because you'd have to add a new column for each new token. The table would also become very sparse—full of zeros—because each individual sentence would only contain a handful of words from the dictionary.

To combat this, it's standard practice to remove common filler words—like the, of, a, is, an, this, etc.—that don't add meaning or differentiation between sentences. These are called stop words. It's also common to remove punctuation and numbers, convert everything to lowercase, and stem words (cut off their endings) to map words like grocery and groceries to the same stem groceri, or reading, read, reads to read. A more advanced counterpart to stemming is called lemmatization, which would map words good, better, best to the root word good. In that sense, lemmatization is “smarter” than stemming, but it takes longer to process.

Little adjustments like this can drastically reduce the size of the dictionary and make analysis easier. Figure 11.2 shows what this process would look like for one sentence.

It should come as no surprise then, after seeing this approach in Figure 11.2, why analyzing text is hard. The process to convert text into numbers has filtered out emotion, context, and word order. If you suspect this would impact the results of any subsequent analysis, you'd be right. And we were lucky here—there aren't any misspellings, which are an added challenge for data workers.

Using the bag-of-words approach, which is available in free software and the first approach data workers learn in text analytics courses, would provide the same numerical encoding for the following two sentences, despite the obvious difference in meaning:

1. Jordan loves hotdogs but hates hamburgers.⁵
2. Jordan hates hotdogs but loves hamburgers.

Humans know the difference between those two sentences. A bag-of-words approach does not. But make no mistake. Despite its simplistic approach, a bag of words can be helpful when summarizing very disparate topics, which we'll highlight in later sections.

N-Grams

It's easy to see where the bag-of-words approach let us down in the example about Jordan and his hotdogs. The two-word phrase “loves hotdogs” is the opposite of “hates hotdogs,” but bag-of-words throws context and word order aside. N-grams can help. An N-gram is a sequence of N consecutive words, so the 2-grams (formally called bigrams) for the sentence “Jordan loves hotdogs but hates hamburgers” would be: {but hates: 1, hates hamburgers: 1, hotdogs but: 1, Jordan loves: 1, loves hotdogs: 1}

It's an extension of bag-of-words but adds the context we need to differentiate phrases with identical words in different arrangements. It's common to add the bigram tokens to the bag-of-words, which further increases the size and sparseness of the document-term matrix. From a practical matter, this means you need to store a large (wide) table with relatively little information inside. We added some bigrams to Table 11.1 to create Table 11.2.

There are some debates whether to filter out bigrams with stop words, as context can be lost. The words “my” and “your” are considered stop words in some software, but the meaning behind the bigram phrases “my preference” and “your preference” is gone if stop words are removed. This is another decision your data workers must make when analyzing text data. (Can you start to sense why text analytics is challenging?)

But once prepared, simple counts can be helpful to summarize text. Websites like Tripadvisor.com take advantage of these approaches and give users the ability to quickly search reviews for frequently mentioned words or phrases. You might see, for example, suggested searches like “baked potato” or “prepared perfectly” as commonly mentioned bigrams at your local steakhouse.

Word Embeddings

With bag-of-words and N-grams, it's possible to tell similarity between documents: if multiple documents contain similar sets of words or N-grams, you can reasonably assume the sentences are related (within reason, of course. Don't forget about Jordan's love/hate relationship with hotdogs). The rows in the document-term matrix would be numerically similar.

But how could someone go about discovering, numerically, which words in a dictionary—not documents, but words—are related?

In 2013, Google analyzed billions of word pairs (two words within close proximity of each other within a sentence) in its enormous database of Google News articles.⁶ By analyzing how often word pairs appeared—for example, (delicious, beef) and (delicious, pork) appeared more than (delicious, cow) and (delicious, pig)—they were able to generate what are called word embeddings, which are numeric representations of words as a list of numbers (aka vectors). If beef and pork appear often with the word delicious, the math would represent each word as being similar in an element of the vector associated with things described as delicious, or what we humans would simply call food.

To explain how it works, we'll use a tiny set of word pairs (Google used billions). Suppose we scan a local newspaper article and find the following pairs of words: (delicious, beef), (delicious, salad), (feed, cow), (beef, cow), (pig, pork), (pork, salad), (salad, beef), (eat, pork), (cow, farm), etc. (Imagine several more pairs along this line of thinking.) Every word goes into the dictionary: {beef, cow, delicious, farm, feed, pig, pork, salad}.⁷

The word cow, for example, can be represented as a vector the length of the preceding dictionary—one component per word with a 1 in the location of cow and zero otherwise: (0, 1, 0, 0, 0, 0, 0, 0). This is an input into a supervised learning algorithm that is mapped to its associated output vector (also the length of the dictionary). In it are the probabilities that the other words in the dictionary appeared near the input word. So for input cow, the associated output might be (0.3, 0, 0, 0.5, 0.1, 0.1, 0, 0), to show cow was paired with beef 30% of the time, farm 50% of the time, and feed & pig 10% of the time.

As is the goal with every supervised learning problem, the model tries to map the inputs (word vectors) as close as possible to the outputs (vectors with probabilities). But here's the twist. We don't care about the model itself; we care about a piece of math generated from the model: a table of numbers that shows how each word in the dictionary relates to every other word. These are the word embeddings. Think of them as a numeric representation of a word that encodes its “meaning.” Table 11.3 shows some of the words in our dictionary, along with their word embeddings across the rows. Cow, for example, is written as a three-dimensional vector (1.0, 0.1, 1.0). Before, it was written as the longer, sparser vector (0, 1, 0, 0, 0, 0, 0, 0).

What's fascinating about word embeddings is that the dimensions (hopefully) contain meaning behind the words, similar in a way to how the reduced dimensions in PCA captured themes of features.

Look down Dimension 1 in Table 11.3. Can you spot a pattern? Whatever Dimension 1 means, Cow and Pig have a lot of it, and Salad has none of it. We might choose to call this Dimension Animal. Dimension 2, we can call Food because Beef, Pork, and Salad scored high, and Dimension 3 Bovine because words associated with cows stood out. With this structure, it's now possible to see similarities in how words are used. It's even possible (if a bit weird) to show simple equations between words.

As an exercise, convince yourself (using Table 11.3) that Beef – Cow + Pig ≈ Pork.⁸

This technique is called Word2vec⁹ (word to vector) and the word embeddings Google generated for us are freely available to download.¹⁰ Of course, don't expect every relationship to be perfect. There's variation in all things, as you're astutely aware of by now, and it's certainly present in text. Non-food items can be described as delicious, like delicious irony. There's orange the color and orange the juice, and a plethora of homonyms in our language to complicate things.

Word embeddings, with their numeric structure that enables calculation, have applications in search engines and recommendation systems, but the word embeddings generated from Google News text might not be specific to your problem set. For example, brand names like Tide® (laundry detergent) and Goldfish® (crackers) may be semantically similar to the words “ocean” and “pet,” respectively, in a system like Word2vec, but for a grocery store, those items would be semantically more similar to competing brands like Gain® detergent and Barnum's Animals® crackers.

It is possible to run Word2vec on your text data and generate your own word embeddings. This would help you glean topics and concepts you might otherwise not even consider in your data. However, getting enough data to be meaningful is an issue. Not every company will have access to as much data as Google. You simply may not have enough text data to find meaningful word embeddings.

Naïve Bayes

The popular method to apply in this situation is a classification algorithm called Naïve Bayes (named after the very same Bayes we mentioned in Chapter 6). The intuition behind it is straightforward: are the words in the email subject line more likely to appear in a spam email or a non-spam email? You probably perform a similar process to this when you read your inbox: the word “free,” based on your experience, is typically a spammy word. As is “money,” “Viagra,” or “rich.” If most words are spammy, the email is probably spam. Easy.

Said another way, you're trying to calculate the probability an email is spam, given the words in the subject line, (w₁, w₂, w_{3  , …}). If this probability is greater than the probability it's not spam, mark the email as spam. In probability notation, these competing probabilities are written as:

• Probability an email is spam = P(spam | w₁, w₂ , w_3 , ...)
• Probability an email is not spam = P(not spam | w₁, w₂ , w₃ , …)

Before moving on, let's take stock of the data available to us in Table 11.4. We have the probability that each word occurs in spam (and not spam) emails. “Free” occurred in three out of the four spam emails, so the probability of seeing “free,” given the email is spam, is P(free | spam) = 0.75. Similar calculations would show: P(debt | spam) = 0.25, P(mom | not spam) = 1, and so on.

What does that give us? We want to know the probability an email is spam given the words, but what we have is the probability of seeing a word, given it's spam. These two probabilities are not the same, but they are linked through Bayes’ Theorem (from Chapter 6). Recall the central idea behind Bayes’ is to swap the conditional probabilities. Thus, instead of working with P(spam | w₁, w₂, w_3  , ...), we can use P(w₁, w₂ , w_3 , … | spam). Through some additional math (which we're skipping for brevity¹⁵), the decision to classify a new email as spam comes down to figuring out which value is higher:

1. Spam score = P(spam) × P(w₁ | spam) × P(w₂ | spam) × P(w₃ | spam)
2. Not Spam score = P(not spam) × P(w₁ | not spam) × P(w₂ | not spam) × P(w₃ | not spam)

All of this information is available in Table 11.4. The probabilities P(spam) and P(not spam) represent the proportion of spam and not spam in the training data—80% and 20%, respectively. In other words, if you had to take a guess without looking at the subject line, you'd guess “spam” because that's the majority class in the training data.

In order to get to the preceding formulas, the Naïve Bayes approach committed what is usually an egregious error in probability—assuming independence between events. The probability of “free” and “Viagra” appearing in a spam email together, denoted P(free, viagra | spam) depends on how often both words appear in the same email, but this makes the computation much more difficult. The “naïve” part of Naïve Bayes is to assume every probability is independent: P(free, viagra | spam) = P(free | spam) × P(viagra | spam).

But there's a slight problem. New and rare words require some adjustments to calculations to avoid multiplying the probabilities by zero. In the tiny dataset in Table 11.4, the word “get” doesn't appear at all, and the words debt, stock, and advice have only shown up in spam. These quirks would make both the spam and not spam scores equal to zero. To fix this, let's pretend we've seen each word at least once by adding 1 to the frequency of each word. We'll also add 2 to the frequency of the spam (and not spam) to prevent values from reaching 1.¹⁶

Now we can calculate:

1. Spam Score: = 0.0074
2. Not Spam Score: = 0.0049

The top number is larger, so we'd predict spam for the email: “Get out of debt with our stock advice!”

Sentiment Analysis

Sentiment analysis is a popular text classification application on social media data. If you search Google for “sentiment analysis of Twitter data,” you might be surprised by the number of results; it seems everyone is doing it. The underlying idea is like the spam/not spam example earlier: are the words in a social media post (or product review, or survey) more likely to be “positive” or “negative.” What you do with this information depends on your business cases, but there is an important callout to mention with sentiment analysis: do not extrapolate beyond the context of the training data and expect meaningful results.

What do we mean? Many “sentiment analysis” classifiers learn from freely available data online. A popular dataset for students is a large collection of movie reviewers from the Internet Movie Database (IMDb.com). This collection of data, and any model you build on it, would be relevant to movie reviews only. Sure, it would associate words like “great” and “awesome” with positive sentiment, but don't expect it to perform well if you have a unique business case with its own vernacular.

Big Tech Has the Upper Hand

Here's what big tech companies like Facebook, Apple, Amazon, Google, and Microsoft have that many other companies don't have: an abundance of text and voice data (an abundance of labeled data that can be used to train supervised learning models); powerful computers; dedicated (and world-class) research teams; and money.

With these resources, they've made remarkable progress not only with text, but also with audio. In recent years, there has been noticeable improvements to the areas of

• Speech-to-Text: Voice-activated assistants and voice-to-text on smartphones are more accurate.
• Text-to-Speech: Computers’ read-aloud voices sound more human-like.
• Text-to-Text: Translating one language to another is instantaneous with good accuracy.
• Chatbots: The automated chat windows that pop up on every website now with: “How may I help you?” are (somewhat) more helpful.
• Generating Human-Readable Text: The language model known as GPT-3¹⁷ from OpenAI is a language model that can generate human-like text (you might think a human wrote it), answer questions, and generate computer code on demand. And it is, as of this writing, the most advanced model of its kind. Estimates put the cost of training the model (not paying the researchers, just running the computers) at $4.6 million.¹⁸

Add to that the access to data and an expert research team, and you get the idea how NLP is often a case of the haves vs. the have nots. Most companies, to be blunt, aren't there (yet?). Even though the algorithms are open source, the massive collection of data and access to supercomputers is not. Big tech clearly has the advantage.

The other thing to consider, when establishing expectations, is how many of the applications from big tech are universal to millions of people; think of them as general tasks common to all sects of society. Amazon Alexa is designed to work for everyone, including children. And when translating text, there are rigid rules built into the training data. The word party in English is the word fiesta in Spanish. Our point is this: everyone who uses these systems is expecting it to work in the same way.

Contrast this to a business-specific text classification task. For example, the sentiment of the sentence “Samsung is better than an iPhone” depends on whether you work for Apple or Samsung. The data you have access to may have its own type of language, unique to only your company. Not only that, but the size of the data will be smaller than what tech companies have available. Consequently, results may not be as clean as you would expect.

Still, we strongly encourage you to avail yourself of all available algorithms including text analytics. Insight is not a matter of big machines, but rather a matter of context and expectation. If you understand the limitations of text analytics before you start, you'll be prepared to run it correctly at your company.