The Requirements

If you look at the website, you can see this is a simple web page with a lot of information. Let me show you which parts we will extract.

One idea would be to extract something from the Halloween-themed site (see Figure 2-1. for their themed landing page). However, this is not an option because you cannot try this yourself; Halloween is over when you read this—at least for 2017, and I cannot guarantee that the future sales will be the same.

Therefore, you will extract information on groceries. To be more specific, you will gather nutrition details from the “Meat & fish” department.

This looks like a lot, but do not worry! You will learn how to extract this information from all the products of this department with an automated script. And if you are keen and motivated, you can extend this knowledge and extract all the nutrition information for all the products.

Preparation

As I mentioned in the previous chapter, before you start your scraper development, you should look at the website’s terms and conditions, and the robots.txt file to see if you can extract the information you need.

When writing this part (November 2017), there was no entry on scraper restrictions in the terms and conditions of the website. This means, you can create a bot to extract information.

In the code block you can see what is allowed and what is not, and this robots.txt is quite restrictive and has only Disallow entries but this is for all bots.

What can we find out from this text? For example, you shouldn’t create bots that order automatically through this website. But this is unimportant for us because we only need to gather information—no purchasing. This robots.txt file has no limitations on our purposes; we are free to continue our preparation and scraping.

Navigating Through “Meat & fishFish”

As mentioned at the beginning of this chapter, we will extract data from the “Meat & fish” department . The URL of this part of the website is www.sainsburys.co.uk/shop/gb/groceries/meat-fish .

Let’s open the URL in our Chrome browser, disable JavaScript, and reload the browser window as described in the previous chapter. Remember, disabling JavaScript enables you to see the website’s HTML code as a basic scraper will see it.

While I am writing this, the website of the department looks like Figure 2-2.

For our purposes, the navigation menu on the left side is interesting. It contains the links to the pages where we will find products to extract. Let’s use the selection tool (or hit CTRL-SHIFT-C) and select the box containing these links, as shown in Figure 2-3.

Now we can see in the DevTools that every link is in a list element (<li> tag) of an unordered list (<ul>), with class categories departments. Note down this information because we will use it later.

Links, which have a little arrow pointing to the right (>), tell us they are just a grouping category and we will find another navigation menu beneath them if we click them. Let’s examine the Roast dinner option , as shown in Figure 2-4.

Here we can see that the page has no products but another list with links to detailed sites. If we look at the HTML structure in DevTools, we can see that these links are again elements of an unordered list. This unordered list has the class categories aisles.

Now we can go further into the Beef category , and here we have products listed (after a big filter box), as shown in Figure 2-5.

Here we need to examine two things: one is the list of products ; the other is the navigation.

If the category contains more products than 36 (this is the default count to show on the website), the items will be split into multiple pages. Because we want to extract information on all products, we must navigate through all those pages. If we select the navigation, we can see it is again an unordered list of the class pages , as shown in Figure 2-6.

From those list elements , we are interested in the one with the right-pointing arrow symbol, which has the class next. This tells us if we have a next page we must navigate to or not.

Now let’s find the link to the detail page of the products. All the products are in an unordered list (again). This list has the class productLister gridView, as shown in Figure 2-7.

Every product is in a list element with the class gridItem. If we open up the details of one of those products we can see where the navigation link is: located in some divs and an h3. We note that the last div has the class productNameAndPromotions , as shown in Figure 2-8.

Now we reached the level of the products, and we can step further and concentrate on the real task: identifying the required information.

Selecting the Required Information

We will discover the elements where our required information resides, based on the product shown in Figure 2-9.

Now that we have the product , let’s identify the required information. As previously, we can use the select tool, locate the required text, and read the properties from the HTML code.

The name of the product is inside a header (h1), which is inside a div with the class productTitleDescriptionContainer.

The price and the unit are in a div of the class pricing. The price itself is in a paragraph (p) of the class pricePerUnit; the unit is in a span of the class pricePerUnitUnit.

Extracting the rating is tricky because here we only see the stars for the rating, but we want the numeric rating itself. Let’s look at the image’s HTML definition, as shown in Figure 2-10.

We can see the location of the image is inside a label of class numberOfReviews and it has an attribute, alt, which contains the decimal value of the averages of the reviews. After the image, there is the text containing the number of the reviews.

The item code is inside a paragraph of class itemCode.

The nutrition information , as shown in Figrue 2-11, is inside a table of class nutritionTable. Every row (tr) of this table contains one entry of our required data: the header (th) of the row has the name and the first column (td) contains the value. The only exception is the energy information, because two rows contain the values but only the first one the header. As you will see, we will solve this problem too with some specific code.

The country of origin, as shown in Figure 2-12, is inside a paragraph of a div of class productText. This field is not unique: every description is in a productText div. This will make the extraction a bit complicated, but there is a solution for this too.

Even though we must extract many fields, we identified them easily in the website. Now it is time to extract the data and learn the tools of the trade!

Outlining the Application

After the requirements are defined and we’ve found each entry to extract, it is time to plan the applications structure and behavior.

And these steps could identify functions of the application we are developing.

Step 1 has a bit more to offer: if you remember the analysis with DevTools you have seen, some links are just a grouping category and you must extract the detail page links from this grouping category.

Navigating the Website

Before we jump into learning the first tools you will use to scrape website data, I want to show you how to navigate websites—and this will be another building block for scrapers.

Websites consist of pages and links between those pages. If you remember your mathematic studies, you will realize a website can be depicted as a graph, as shown in Figure 2-13.

Because a website is a graph, you can use graph algorithms to navigate through the pages and links: Breadth First Search (BFS) and Depth First Search (DFS).

Using BFS, you go one level of the graph and gather all the URLs you need for the next level. For example, you start at the “Meat & fish” department page and extract all URLs to the next required level, like “Top sellers” or “Roast dinner.” Then you have all these URLs and go to the Top sellers and extract all URLs that lead to the detailed product pages. After this is done, you go to the “Roast dinner” page and extract all product details from there too, and so on. At the end you will have the URLs to all product pages, where you can go and extract the required information.

Using DFS, you go straight to the first product through “Meat & fish,” “Top sellers,” and extract the information from its site. Then you go to the next product on the “Top sellers” page and extract the information from there. If you have all the products from “Top sellers” then you move to “Roast dinner” and extract all products from there.

If you ask me, both algorithms are good, and they deliver the same result. I could write two scripts and compare them to see which one is faster, but this comparison would be biased and flawed.¹

Therefore, you will implement a script that will navigate a website, and you can change the algorithm behind it to use BFS or DFS.

If you are interested in the Why? for both algorithms, I suggest you consider Magnus Hetland’s book: Python Algorithms.²

Summary

This chapter prepared you for the remaining parts of the book: you’ve met the requirements, analyzed the website to scrape, and identified where in the HTML code the fields of interest lay. And you implemented a simple scraper, mostly with basic Python tools, which navigates through the website.

In the next chapter you will learn Beautiful Soup, a simple extractor library that helps you to forget regular expressions, and adds more features to traverse and extract HTML-trees like a boss.