Parsing the data file

The data file contains complex data to track user behavior, and you may encounter this sort of data when performing data science tasks. It looks complicated at first, but if you break the data file down carefully, you can eventually tease out the file details. If you were to open this data file (it’s text, so you can look if desired), you would find that it contains three kinds of records:

• A: Attributes of the particular page. Each attribute is a different page, so you could use the word page (or pages for multiples) in place of attributes, but the example uses attributes for clarity.
• C: Users who are looking at the pages.
• V: Vroots for each of the pages. A Vroot is a series of grouped website pages. Together they constitute an area of the website. The binary values show whether someone has visited a certain area. (You just see a flag; you don’t see how many times the user has actually visited that website area.)

Each record appears on a separate line. Consequently, you build one dictionary for each of the record types to separate one from the other, as shown here:

import codecs

import collections

# Open the file.

file = codecs.open(filename, 'r')

# Setup for attributes.

attribute = collections.namedtuple(

'page', ['id', 'description', 'url'])

attributes = {}

# Setup for users

current_user_id = None

current_user_ids = []

user_visits = {}

# Setup for Vroots

page_visits = {}

# Process the data one line at a time and place

# each record in the appropriate storage unit.

for line in file:

chunks = line.split(',')

entry_type = chunks[0]

if entry_type == 'A':

type, id, ignored, description, url = chunks

attributes[int(id)] = attribute(

id=int(id), description=description, url=url)

if entry_type == 'C':

if not current_user_id == None:

user_visits[current_user_id] = set(

current_user_ids)

current_user_ids = []

current_user_id = int(chunks[2])

if entry_type == 'V':

page_id = int(chunks[1])

current_user_ids.append(page_id)

page_visits.setdefault(page_id, [])

page_visits[page_id].append(current_user_id)

# Display the totals

print('Total Number of Attributes: ',

len(attributes.keys()))

print('Total Number of Users: ', len(user_visits.keys()))

print('Total Number of VRoots: ', len(page_visits.keys()))

The code begins by setting up variables to hold information for each of the record types. It then reads the file one line at a time and determines the record type. Each record requires a different kind of process. For example, an attribute contains a page number, description, and URL. User records contain the user ID and a list of pages that the user has visited. The Vroot entries associate pages with users. At the end of the process, you can see the number of each kind of record in the dataset.

Total Number of Attributes: 294

Total Number of Users: 32710

Total Number of VRoots: 285

The idea is that a user’s visit to a certain area indicates a specific interest. For instance, when a user visits pages to learn about productivity software along with visits to a page containing terms and prices, this behavior indicates an interest in acquiring the productivity software soon. Useful recommendations can be based on such inferences about a user’s desire to buy certain versions of the productivity software or bundles of different software and services.

Viewing the attributes

It’s important to remember that the focus is on pages and users viewing them, so it pays to know a little something about the pages. After you parse the dataset, the following code will display the page information for you:

for k, v in attributes.items():

print("{:4} {:30.30} {:12}".format(

v.id, v.description, v.url))

When you run this code, you see all 294 attributes (pages). Here is a partial listing:

1287 "International AutoRoute" "/autoroute"

1288 "library" "/library"

1289 "Master Chef Product Infor…" "/masterchef"

1297 "Central America" "/centroam"

1215 "For Developers Only Info" "/developer"

1279 "Multimedia Golf" "/msgolf"

1239 "Microsoft Consulting" "/msconsult"

Obtaining statistics

In addition to viewing the data, you can also perform analysis on it by various means, such as statistics. Here are some statistics you can try with the users:

nbr_visits = list(map(len, user_visits.values()))

average_visits = sum(nbr_visits) / len(nbr_visits)

one_visit = sum(x == 1 for x in nbr_visits)

print("Number of user visits: ", sum(nbr_visits))

print("Average number of visits: ", average_visits)

print("Users with just one visit: ", one_visit)

When you run this code, you see some interesting information about the users who visited the various pages:

Number of user visits: 98653

Average number of visits: 3.0159889941913787

Users with just one visit: 9994

Considering collaborative filtering

When using collaborative filtering, you need to calculate similarity. See Chapter 14 of Machine Learning For Dummies, by John Paul Mueller and Luca Massaron (Wiley), for a discussion of the use of similarity measures. Another good place to look is at http://dataaspirant.com/2015/04/11/five-most-popular-similarity-measures-implementation-in-python/. Apart from Euclidean, Manhattan, and Chebyshev distances, the remainder of this section discusses cosine similarity. Cosine similarity measures the angular cosine distance between two vectors, which may seem like a difficult concept to grasp but is just a way to measure angles in data spaces.

The idea behind the cosine distance is to use the angle created by the two points connected to the space origin (the point where all dimensions are zero) instead. If the points are near, the angle is narrow, no matter how many dimensions are there. If they are far away, the angle is quite large. Cosine similarity implements the cosine distance as a percentage and is quite effective in telling whether a user is similar to another or whether a film can be associated to another because the same users favor it.

Obtaining the data

The code in this section assumes that you have access to the MovieLens database using the code from the “Using the MovieLens sparse matrix” section of Book 4, Chapter 4. Assuming that you’re working with a new notebook, however, you need to read the data into the notebook and merge the two datasets used for this example, as shown here:

import pandas as pd

ratings = pd.read_csv("ml-20m/ratings.csv")

movies = pd.read_csv("ml-20m/movies.csv")

movie_data = pd.merge(ratings, movies, on="movieId")

print(movie_data.head())

After you perform the merge, you see a new dataset, movie_data, which contains the combination of ratings and movies, as shown here:

userId movieId rating timestamp title

0 1 2 3.5 1112486027 Jumanji (1995)

1 5 2 3.0 851527569 Jumanji (1995)

2 13 2 3.0 849082742 Jumanji (1995)

3 29 2 3.0 835562174 Jumanji (1995)

4 34 2 3.0 846509384 Jumanji (1995)

genres

0 Adventure|Children|Fantasy

1 Adventure|Children|Fantasy

2 Adventure|Children|Fantasy

3 Adventure|Children|Fantasy

4 Adventure|Children|Fantasy

All these entries are for Jumanji because head() shows only the first five entries in the movie_data dataset, and Jumanji obviously has at least five ratings. You can use the new dataset to obtain a simple statistic for the movies; however, the mean of the ratings for each movie is shown here:

print(movie_data.groupby('title')['rating'].mean().head())

This code looks rather complicated, but it isn't. Calling groupby('title') creates a grouping of the various movies by title. You can then access the ['rating'] column of that grouping to obtain a mean(). The output shows the first five entries, as shown here (note that groupby() automatically sorts the entries for you):

title

"Great Performances" Cats (1998) 2.748387

#chicagoGirl: The Social Network Takes on a… 3.666667

$ (Dollars) (1971) 2.833333

$5 a Day (2008) 2.871795

$9.99 (2008) 3.009091

Name: rating, dtype: float64

The rating column doesn't have a title, but you see it listed on the last line as the column used to create the mean, which is of type float64.

Massaging the data

The current MovieLens dataset is huge and cumbersome. When working with an online product, such as Google Colab (see Book 1, Chapter 3 for details), the dataset might very well work in its current form. When working with a desktop system, you need to massage the data to ensure that you actually can get the desired results. In fact, massaging the data is an essential part of performing data science tasks because you may not actually have good data. This section looks at ways that you might want to massage the MovieLens dataset to ensure good results.

Desktop setups can be particularly picky when you’re working with huge data. One of the issues you can encounter when working with these datasets is memory. When performing certain tasks, such as creating the pivot table for this example, you might see ValueError: negative dimensions are not allowed as an output. What this really means is that your system ran out of memory. You have a number of options for countering this problem, some of the most important of which appear in this section.

You can reduce the memory requirements for working with the data by removing items that you don't really want in the analysis anyway. For this analysis, you have three extra columns: movieId, timestamp, and genres. In addition, a person would need to think enough of a movie to give it at least three out of five stars. Consequently, you can also get rid of the lesser value reviews using the following code:

reduced_movie = movie_data.loc[

movie_data['rating'] >= 3.0]

reduced_movie = reduced_movie.drop(

columns=['movieId','timestamp', 'genres'])

print(reduced_movie.head())

print()

print("Original Shape: {0}, New Shape: {1}".format(

movie_data.shape, reduced_movie.shape))

The reduction in size doesn’t actually affect the better movies. Instead, you just lose lesser movies that would have unfavorably affected the results. The size of the reduced_movie dataset is significantly smaller than the original movie_data dataset, as shown here:

userId rating title

0 1 3.5 Jumanji (1995)

1 5 3.0 Jumanji (1995)

2 13 3.0 Jumanji (1995)

3 29 3.0 Jumanji (1995)

4 34 3.0 Jumanji (1995)

Original Shape: (20000263, 6), New Shape: (16486759, 3)

The number of reviews also reflects the popularity of a movie. When a movie has few reviews, it might reflect a cult following — a group of devotees who don’t reflect the opinion of the public at large. You can remove movies with only a few reviews using the following code:

reduced_movie = reduced_movie[

reduced_movie.groupby('title')['rating'].transform(

'size') > 3000]

print(reduced_movie.groupby('title')[

'rating'].count().sort_values().head())

print()

print("New shape: ", reduced_movie.shape)

The call to transform() selects only movies that have a certain number of reviews — more than 3,000 of them in this case. You can use transform() in a huge number of ways based solely on the function you provide as input, which is the built-in size function in this case. Here is the result of this particular bit of trimming:

title

Eastern Promises (2007) 3001

Triplets of Belleville, The (Les triplettes de Bel… 3003

Bad Santa (2003) 3006

Mexican, The (2001) 3010

1984 (Nineteen Eighty-Four) (1984) 3010

Name: rating, dtype: int64

New shape: (12083404, 3)

The way you shape your data will affect the output of any analysis you perform. You may not get the desired results the first time, so you may end up spending a lot of time trying different shaping methods. The point is to keep trying to shape the data in various ways until you obtain a good result.

A final way to save memory for analysis purposes is to clean up your variables, which can consume a lot of memory. This example uses the following code for this purpose:

ratings = None

movies = None

movie_data = None

Performing collaborative filtering

Making recommendations depends on finding the right kind of information on which to make a comparison. Of course, this is where the art of data science comes into play. If making a recommendation only involved performing analysis on data in a particular manner using a specific algorithm, anyone could do it. The art is in choosing the correct data to analyze. In this section, you use a combination of the user ID and the ratings assigned by those users to a particular movie as the means to perform collaborative filtering. In other words, you’re making an assumption that people who have similar tastes in movies will rate those movies at a particular level.

After you’ve shaped your data, you can use it to create a pivot table. The pivot table will compare user IDs with the reviews that the user has created for particular movies. Here is the code used to create the pivot table:

user_rating = pd.pivot_table(

reduced_movie,

index='userId',

columns='title',

values='rating')

print(user_rating.head())

The results might look a little odd because the pivot table will be a sparse matrix like the sample shown here:

title Young Frankenstein Young Guns Zodiac

userId

1 4.0 NaN NaN

2 NaN NaN NaN

3 5.0 NaN NaN

4 NaN NaN NaN

5 NaN NaN NaN

In this case, you see that Young Frankenstein is the only movie that was rated by users 1 through 5. The point is that the rows contain individual user reviews and the columns are the names of movies they reviewed.

The next step in the process is to obtain a listing of reviews for the target movie, which is Young Frankenstein. The following code creates a list of reviewers:

YF_ratings = user_rating['Young Frankenstein (1974)']

print(YF_ratings.sort_values(ascending=False).head())

The output of this part of the code shows that Jumanji isn’t the most popular movie around, but it’ll work for the example:

userId

60898 5.0

52548 5.0

101177 5.0

101198 5.0

28648 5.0

Name: Young Frankenstein (1974), dtype: float64

Now that you have sample data to use, you can correlate it with the pivot table as a whole. The following code outputs the movies that most closely match Jumanji in appeal by the users who liked Jumanji:

print(user_rating.corrwith(

YF_ratings).sort_values(

ascending=False).head())

The output shows that you can derive some interesting results using collaborative filtering techniques:

title

Young Frankenstein (1974) 1.000000

Blazing Saddles (1974) 0.421143

Monty Python and the Holy Grail (1975) 0.300413

Producers, The (1968) 0.297317

Magnificent Seven, The (1960) 0.291847

dtype: float64

Even though the correlation results seem a little low (with 1.000000 being the most desirable), the names of the movies selected make sense. For example, like Young Frankenstein, Blazing Saddles is a Mel Brooks movie, and Monty Python and the Holy Grail is a comedy.