Word Frequency Analysis

As usual, you should at least attempt the exercises before you read my solutions.

Random Numbers

Given the same inputs, most computer programs generate the same outputs every time, so they are said to be deterministic. Determinism is usually a good thing, since we expect the same calculation to yield the same result. For some applications, though, we want the computer to be unpredictable. Games are an obvious example, but there are more.

Making a program truly nondeterministic turns out to be difficult, but there are ways to make it at least seem nondeterministic. One of them is to use algorithms that generate pseudorandom numbers. Pseudorandom numbers are not truly random because they are generated by a deterministic computation, but just by looking at the numbers it is all but impossible to distinguish them from random.

The function rand returns a random float between 0.0 and 1.0 (including 0.0 but not 1.0). Each time you call rand, you get the next number in a long series. To see a sample, run this loop:

for i in 1:10
    x = rand()
    println(x)
end

The function rand can take an iterator or array as an argument and return a random element:

for i in 1:10
    x = rand(1:6)
    print(x, " ")
end

julia> t = ['a', 'a', 'b'];

julia> histogram(t)
Dict{Any,Any} with 2 entries:
  'a' => 2
  'b' => 1

Word Histogram

You should attempt the previous exercises before you go on. You will also need the emma.txt file available from this book’s GitHub repository.

Here is a program that reads a file and builds a histogram of the words in the file:

function processfile(filename)
    hist = Dict()
    for line in eachline(filename)
        processline(line, hist)
    end
    hist
end;

function processline(line, hist)
    line = replace(line, '-' => ' ')
    for word in split(line)
        word = string(filter(isletter, [word...])...)
        word = lowercase(word)
        hist[word] = get!(hist, word, 0) + 1
    end
end;

hist = processfile("emma.txt");

This program reads emma.txt, which contains the text of Emma by Jane Austen.

processfile loops through the lines of the file, passing them one at a time to processline. The histogram hist is being used as an accumulator.

processline uses the function replace to replace hyphens with spaces before using split to break the line into an array of strings. It traverses the array of words and uses filter, isletter, and lowercase to remove punctuation and convert to lower case. (It is shorthand to say that strings are “converted”; remember that strings are immutable, so a function like lowercase returns new strings.)

Finally, processline updates the histogram by creating a new item or incrementing an existing one.

To count the total number of words in the file, we can add up the frequencies in the histogram:

function totalwords(hist)
    sum(values(hist))
end

The number of different words is just the number of items in the dictionary:

function differentwords(hist)
    length(hist)
end

Here is some code to print the results:

julia> println("Total number of words: ", totalwords(hist))
Total number of words: 162742
julia> println("Number of different words: ", differentwords(hist))
Number of different words: 7380

Most Common Words

To find the most common words, we can make an array of tuples, where each tuple contains a word and its frequency, and sort it. The following function takes a histogram and returns an array of word-frequency tuples:

function mostcommon(hist)
    t = []
    for (key, value) in hist
        push!(t, (value, key))
    end
    reverse(sort(t))
end

In each tuple, the frequency appears first, so the resulting array is sorted by frequency. Here is a loop that prints the 10 most common words:

t = mostcommon(hist)
println("The most common words are:")
for (freq, word) in t[1:10]
    println(word, "	", freq)
end

I use a tab character (' ') as a “separator,” rather than a space, so the second column is lined up. Here are the results from Emma:

The most common words are:
to  5295
the 5266
and 4931
of  4339
i   3191
a   3155
it  2546
her 2483
was 2400
she 2364

Optional Parameters

We have seen built-in functions that take optional arguments. It is possible to write programmer-defined functions with optional arguments, too. For example, here is a function that prints the most common words in a histogram:

function printmostcommon(hist, num=10)
    t = mostcommon(hist)
    println("The most common words are: ")
    for (freq, word) in t[1:num]
        println(word, "	", freq)
    end
end

The first parameter is required; the second is optional. The default value of num is 10.

If you only provide one argument:

printmostcommon(hist)

num gets the default value. If you provide two arguments:

printmostcommon(hist, 20)

num gets the value of the argument instead. In other words, the optional argument overrides the default value.

If a function has both required and optional parameters, all the required parameters have to come first, followed by the optional ones.

Dictionary Subtraction

Finding the words from a book that are not in the word list from words.txt is a problem you might recognize as set subtraction; that is, we want to find all the words from one set (the words in the book) that are not in the other (the words in the list).

subtract takes dictionaries d1 and d2 and returns a new dictionary that contains all the keys from d1 that are not in d2. Since we don’t really care about the values, we set them all to nothing:

function subtract(d1, d2)
    res = Dict()
    for key in keys(d1)
        if key ∉ keys(d2)
            res[key] = nothing
        end
    end
    res
end

To find the words in the book you downloaded that are not in words.txt, you can use processfile to build a histogram for words.txt, and then subtract:

words = processfile("words.txt")
diff = subtract(hist, words)

println("Words in the book that aren't in the word list:")
for word in keys(diff)
    print(word, " ")
end

Here are some of the results from Emma:

Words in the book that aren't in the word list:
outree quicksighted outwardly adelaide rencontre jeffereys unreserved dixons betweens ...

Some of these words are names and possessives. Others, like “rencontre,” are no longer in common use. But a few are common words that should really be in the list!

Random Words

To choose a random word from the histogram, the simplest algorithm is to build an array with multiple copies of each word, according to the observed frequency, and then choose from the array:

function randomword(h)
    t = []
    for (word, freq) in h
        for i in 1:freq
            push!(t, word)
        end
    end
    rand(t)
end

This algorithm works, but it is not very efficient; each time you choose a random word it rebuilds the array, which is as big as the original book. An obvious improvement is to build the array once and then make multiple selections, but the array is still big.

An alternative is:

1. Use keys to get an array of the words in the book.

2. Build an array that contains the cumulative sum of the word frequencies (see “Exercise 10-2”). The last item in this array is the total number of words in the book, $n$.

3. Choose a random number from 1 to $n$. Use a bisection search (see “Exercise 10-10”) to find the index where the random number would be inserted in the cumulative sum.

4. Use the index to find the corresponding word in the word array.

Markov Analysis

If you choose words from the book at random, you can get a sense of the vocabulary, but you probably won’t get a sentence:

this the small regard harriet which knightley's it most things

A series of random words seldom makes sense because there is no relationship between successive words. For example, in a real sentence you would expect an article like “the” to be followed by an adjective or a noun, and probably not a verb or adverb.

One way to measure these kinds of relationships is Markov analysis, which characterizes, for a given sequence of words, the probability of the words that might come next. For example, the song “Eric, the Half a Bee” (by Monty Python) begins:

Half a bee, philosophically,
Must, ipso facto, half not be.
But half the bee has got to be
Vis a vis, its entity. D’you see?

But can a bee be said to be
Or not to be an entire bee
When half the bee is not a bee
Due to some ancient injury?

In this text, the phrase “half the” is always followed by the word “bee,” but the phrase “the bee” might be followed by either “has” or “is.”

The result of Markov analysis is a mapping from each prefix (like “half the” and “the bee”) to all possible suffixes (like “has” and “is”).

Given this mapping, you can generate random text by starting with any prefix and choosing at random from the possible suffixes. Next, you can combine the end of the prefix and the new suffix to form the next prefix, and repeat.

For example, if you start with the prefix “Half a,” then the next word has to be “bee,” because the prefix only appears once in the text. The next prefix is “a bee,” so the next suffix might be “philosophically,” “be,” or “due.”

In this example the length of the prefix is always 2, but you can do Markov analysis with any prefix length.

Data Structures

Using Markov analysis to generate random text is fun, but there is also a point to this exercise: data structure selection. In completing the previous exercise, you had to choose:

• How to represent the prefixes

• How to represent the collection of possible suffixes

• How to represent the mapping from each prefix to the collection of possible suffixes

The last one is easy: a dictionary is the obvious choice for a mapping from keys to corresponding values.

For the prefixes, the most obvious options are a string, array of strings, or tuple of strings.

For the suffixes, one option is an array; another is a histogram (dictionary).

How should you choose? The first step is to think about the operations you will need to implement for each data structure. For the prefixes, you need to be able to remove words from the beginning and add to the end. For example, if the current prefix is “Half a,” and the next word is “bee,” you need to be able to form the next prefix, “a bee.”

Your first choice might be an array, since it is easy to add and remove elements.

For the collection of suffixes, the operations you need to perform include adding a new suffix (or increasing the frequency of an existing one) and choosing a random suffix.

Adding a new suffix is equally easy for the array implementation or the histogram. Choosing a random element from an array is easy; choosing from a histogram is harder to do efficiently (see “Exercise 13-7”).

So far we have been talking mostly about ease of implementation, but there are other factors to consider in choosing data structures. One is runtime. Sometimes there is a theoretical reason to expect one data structure to be faster than another; for example, I mentioned that the in operator is faster for dictionaries than for arrays, at least when the number of elements is large.

But often you don’t know ahead of time which implementation will be faster. One option is to implement both of them and see which is better. This approach is called benchmarking. A practical alternative is to choose the data structure that is easiest to implement, and then see if it is fast enough for the intended application. If so, there is no need to go on. If not, there are tools, like the Profile module, that can identify the places in a program that take the most time.

The other factor to consider is storage space. For example, using a histogram for the collection of suffixes might take less space because you only have to store each word once, no matter how many times it appears in the text. In some cases, saving space can also make your program run faster, and in the extreme, your program might not run at all if you run out of memory. But for many applications, space is a secondary consideration after runtime.

One final thought: in this discussion, I have implied that you should use one data structure for both analysis and generation. But since these are separate phases, it would also be possible to use one structure for analysis and then convert to another structure for generation. This would be a net win if the time saved during generation exceeded the time spent in conversion.

Glossary

deterministic

Pertaining to a program that does the same thing each time it runs, given the same inputs.

pseudorandom

Pertaining to a sequence of numbers that appears to be random, but is generated by a deterministic program.

default value

The value given to an optional parameter if no argument is provided.

override

To replace a default value with an argument.

benchmarking

The process of choosing between data structures by implementing alternatives and testing them on a sample of the possible inputs.

rubberducking

Debugging by explaining your problem to an inanimate object such as a rubber duck. Articulating the problem can help you solve it, even if the rubber duck doesn’t know Julia.

Exercises

(v1.0) pkg> add Plots

using Plots
x = 1:10
y = x.^2
plot(x, y)