Projects for Website Scraping

There are a lot of use cases where you can leverage your knowledge of website scraping. Some might be common sense, while others are extreme cases. In this section you will find some use cases where you can leverage your knowledge.

The main reason to create a scraper is to extract information from a website. This information can be a list of products sold by a company, nutrition details of groceries, or NFL results from the last 15 years. Most of these projects are the groundwork for further data analysis: gathering all this data manually is a long and error-prone process.

Sometimes you encounter projects where you need to extract data from one website to load it into another—a migration. I recently had a project where my customer moved his website to WordPress and the old blog engine’s export functionality wasn’t meant to import it into WordPress. I created a scraper that extracted all the posts (around 35,000) with their images, did some formatting on the contents to use WordPress short codes, and then imported all those posts into the new website.

A weird project could be to download the whole Internet! Theoretically it is not impossible: you start at a website, download it, extract and follow all the links on this page, and download the new sites too. If the websites you scrape all have links to each other, you can browse (and download) the whole Internet. I don’t suggest you start this project because you won’t have enough disk space to contain the entire Internet, but the idea is interesting. Let me know how far you reached if you implement a scraper like this.

Websites Are the Bottleneck

One of the most difficult parts of gathering data through websites is that websites differ. I mean not only the data but the layout too. It is hard to create a good-fit-for-all scraper because every website has a different layout, uses different (or no) HTML IDs to identify fields, and so on.

And if this is not enough, many websites change their layout frequently. If this happens, your scraper is not working as it did previously. In these cases, the only option is to revisit your code and adapt it to the changes of the target website.

Unfortunately, you won’t learn secret tricks that will help you create a scraper that always works—if you want to write specialized data extractors. I will show some examples in this book that will always work if the HTML standard is in use.

Terms and Robots

Scraping currently has barely any limitations; there are no laws defining what can be scraped and what cannot.

However, there are guidelines that define what you should respect. There is no enforcing; you can completely ignore these recommendations, but you shouldn’t.

Before you start any scraping task, look at the Terms & Conditions and Privacy Policy of the website you want to gather data from. If there is no limitation on scraping, then you should look at the robots.txt file for the given website(s).

Most of the time these keywords can be found, and this makes your search easier. If you have no luck, you need to read through the whole legal content and it is not as easy—at least I think legal stuff is always dry to read.

In the European Union there’s a data protection right that has been live for some years but strictly enforced from 2018: GDPR. Keep the private data of private persons out of your scraping—you can be held liable if some of it slips out into public because of your scraper.

Technology of the Website

Another useful preparation step is to look at the technologies the targeted website uses.

There is a Python library called builtwith , which aims to detect the technologies a website utilizes. The problem with this library is that the last version 1.3.2 was released in 2015, and it is not compatible with Python 3. Therefore, you cannot use it as you do with libraries available from the PyPI.¹

However, in May 2017, Python 3 support has been added to the sources, but the new version was not released (yet, I’m writing this in November 2017). This doesn’t mean we cannot use the tool; we must manually install it.

First, download the sources from https://bitbucket.org/richardpenman/builtwith/downloads/ . If you prefer, you can clone the repository with Mercurial to stay up to date if new changes occur.

After downloading the sources, navigate to the folder where you downloaded the sources and execute the following command:

pip install .

The command installs builtwith to your Python environment and you can use it.

The preceding code block shows which technologies Apress uses for its website. You can learn from AngularJS that if you plan to write a scraper, you should be prepared to handle dynamic content that is rendered with JavaScript.

builtwith is not a magic tool, it is a website scraper that downloads the given URL; parses its contents; and based on its knowledge base, it tells you which technologies the website uses. This tool uses basic Python features, which means sometimes you cannot get information in the website you are interested in, but most of the time you get enough information.

Using Chrome Developer Tools

As you can see, DevTools give you tools to see inside the workings of the browser. We don’t need anything special; we will use DevTools to see where the information resides.

In this section I will guide us with screenshots through the steps I usually do when I start (or just evaluate) a scraping project.

Set-up

First, you must prepare to get the information. Even though we know which website to scrape and what kind of data to extract, we need some preparation.

Basic website scrapers are simple tools that download the contents of the website into memory and then do extraction on this data. This means they are not capable of running dynamic content just like JavaScript, and therefore we have to make our browser similar to a simple scraper by disabling JavaScript rendering.

First, right-click with your mouse on the web page and from the menu select “Inspect,” as shown in Figure 1-1.

Alternatively, you can press CTRL+SHIFT+I in Windows or z+⇧+I on a Mac to open the DevTools window.

Then locate the settings button (the three vertically aligned dots , as shown in Figure 1-2.) and click it:

Alternatively, you can press F1 in Windows.

Now scroll down to the bottom of the Settings screen and make sure Disable JavaScript is checked, as shown in Figure 1-3.

Now reload the page, exit the Settings window, but stay in the inspector view because we will use the HTML element selector available here.

Note

Disabling JavaScript is necessary if you want to see how your scraper sees the website.

Later in this book, you will learn options how to scrape websites that utilize JavaScript to render dynamic content.

But to fully understand and enjoy those extra capabilities, you must learn the basics.

Tool Considerations

If you are reading this book, you will write your scrapers most likely with Python 3. However, you must decide on which tools to use.

In this book you will learn the tools of the trade and you can decide on your own what to use, but now I’ll share with you how I decide on an approach.

If you are dealing with a simple website—and by simple, I mean one that is not using JavaScript excessively for rendering—then you can choose between creating a crawler with Beautiful Soup + requests or use Scrapy. If you must deal with a lot of data and want to speed things up, use Scrapy. In the end, you will use Scrapy in 90% of your tasks, and you can integrate Beautiful Soup into Scrapy and use them together.

If the website uses JavaScript for rendering, you can either reverse engineer the AJAX/XHR calls and use your preferred tool, or you can reach out to a tool that renders websites for you. Such tools are Selenium and Portia . I will introduce you to these approaches in this book and you can decide which fits you best, which is easier for you to use.

Parsing robots.txt

Let’s create an application that parses the robots.txt file of the target website and acts based on the contents.

Python has a built-in module that is called robotparser, which enables us to read and understand the robots.txt file and ask the parser if we can scrape a given part of the target website.

This code snippet is good if you write your own scraper and you don’t use Scrapy. Integrating the robotparser and checking every URL before accessing it helps you automate the task of honoring the website owners’ request what to access.

Unfortunately, you cannot prevent malicious bots from scraping your website because in most cases they will ignore the settings in your robots.txt file.

Creating a Link Extractor

After this lengthy introduction, it is time to create our first scraper, which will extract links from a given page.

This example will be simple; we won’t use any specialized tools for website scraping, just libraries available with the standard Python 3 installation.

As you can see, when you run the modified code, this new method adds http://www.apress.com to every URL that is missing this prefix, for example http://www.apress.com/gp/python , but leaves others like https://twitter.com/apress intact.

The previous code example uses regular expressions to find all the anchor tags (<a>) in the HTML code of the website. Regular expressions are a hard topic to learn, and they are not easy to write. That’s why we won’t dive deeper into this topic and will use more high-level tools, like Beautiful Soup, in this book to extract our contents.

Extracting Images

In this section we will extract image sources from the website. We won’t download any images yet, just lay hands on the information about where these images are in the web.

Images are very similar to links from the previous section, but they are defined by the <img> tag and have a src attribute instead of an href.

If you take a close look, I modified just some variable names and the regular expression. I could have used the link extractor from the previous section and changed only the expression.