Download Data from a Web Server

When you type an address into your web browser and hit Enter, your browser is the client. It establishes a connection with the server, which responds with a web page. Of course, a client doesn’t have to be a web browser; it can also be a mail application, a weather widget on your phone or computer, or a game that uploads your high score to a global leaderboard. In the first part of this chapter, we’re going to focus on projects that use the Raspberry Pi to act as a client. The code you’ll be using will connect to Internet servers to get information. Before you can do that, you’ll need to install a popular Python library called Requests that is used for connecting to web servers via  hypertext transfer protocol, or HTTP.

In order to use Requests in Python, you first need to import it. Within Terminal:

$ python
Python 2.7.3rc2 (default, May  6 2012, 20:02:25)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more 
  information.
>>> import requests
>>>

If you don’t get any kind of error message, you’ll know Requests has been imported in this Python session.

Now you can try it out:

>>> r = requests.get('http://www.google.com/')
>>>

You may be a bit disappointed at first because it seems like nothing happened. But actually, all the data from the request has been stored in the object r. Here’s how you can display the status code:

>>> r.status_code
200

The HTTP status code 200 means that the request succeeded. There are a few other HTTP status codes in Table 10-1.

If you want to see the contents of the response (what the server sends back to you), try the following:

>>> r.text

If everything worked correctly, what will follow is a large block of text; you may notice some human-readable bits in there, but most of it will be hard to understand. This is the HTML of Google’s landing page, which is meant to be interpreted and rendered on screen by a web browser.

However, not all HTTP requests are meant to be rendered by a web browser. Sometimes only data is transmitted, with no information about how it should be displayed. Many sites make these data protocols available to the public so that we can use them to fetch data from and send data to their servers without using a web browser. Such a data protocol specification is commonly called an application programming interface, or API. APIs let different pieces of software talk to each other and are popular for sending data from one site to another over the Internet.

For example, let’s say you want to make a project that will sit by your door and remind you to take your umbrella with you when rain is expected that day. Instead of setting up your own weather station and figuring out how to forecast the precipitation, you can get the day’s forecast from one of many weather APIs out there.

Fetching the Weather Forecast

In order to determine whether or not it will rain today, we’ll show you how to use the API from Weather Underground.

To use the API, take the following steps:

1. In a web browser, go to Weather Underground’s API homepage and enter your information to sign up.

2. After you’ve logged in to your account, click Key Settings.

3. Notice that there’s a long string of characters in the API key field (Figure 10-1). This is a unique identifier for your account that you’ll need to provide in each request. If you abuse its servers by sending too many requests, it can simply shut off this API key and refuse to complete any further requests until you pay for a higher tier of service.

4. Click Documentation and then Forecast so you can see what kind of data you get when you make a forecast request. At the bottom of the page is an example URL for getting the forecast for San Francisco, CA. Notice that in the URL is your API key:

http://api.wunderground.com/api/YourAPIkey/forecast/q/CA/San_Francisco.json

Figure 10-1. Matt’s Weather Underground API account, with the API key in the upper right

5. To make sure it works, go to this URL into your web browser, and you should see the weather forecast data in a format called JSON, or JavaScript Object Notation (see Example 10-1). Notice how the data is structured hierarchically.

Note

Even though the J stands for JavaScript, JSON is used in many programming languages, especially for communicating between applications via an API.

6. To start off, try to get today’s local text forecast from within this data structure. Change the URL so that it matches your state and town and put it in a new Python script called text-forecast.py (Example 10-2).

7. As you see from the code, in order to get the text forecast for today, we’ll have to fetch the right field from the hierarchy of the data structure (see Example 10-1). The text forecast can be found in forecast→txt_forecast→forecastday→0 (the first entry, which is today’s forecast)→fcttext.

8. When you run the script from the command line, the output should be your local weather forecast for today, such as this: “Clear. High of 47F. Winds from the NE at 5 to 10 mph.”

Note

Keep in mind that not all APIs are created equal and you’ll have to review their documentation to determine if it’s the right one for your project. Also, most APIs limit the number of requests you can make, and some even charge you to use their services. Many times, the API providers have a free tier for a small amount of daily requests, which is perfect for experimentation and personal use.

Example 10-1. Partial JSON response from Weather Underground’s API

"response": {
  "version": "0.1",
  "termsofService":
    "http://www.wunderground.com/weather/api/d/terms.html",
  "features": {
    "forecast": 1
  }
},
"forecast":{ 
  "txt_forecast": { 
    "date":"10:00 AM EST",
    "forecastday": [ 
      {
      "period":0,
      "icon":"partlycloudy",
      "icon_url":"http://icons-ak.wxug.com/i/c/k/
        partlycloudy.gif",
      "title":"Tuesday",
      "fcttext":
        "Partly cloudy. High of 48F. Winds from the NNE at 5 to 
          10 mph.", 
      "fcttext_metric":
        "Partly cloudy. High of 9C. Winds from the NNE at 10 to 
          15 km/h.",
      "pop":"0"
      },

Forecast is the top-level parent of the data we want to access.

Within forecast, we want the text forecast.

Within the text forecast dataset, we want the daily forecasts.

fcttext, the text of the day’s forecast.

Example 10-2. Source code for text-forecast.py

import requests

key = 'YOUR KEY HERE' 
ApiUrl = 'http://api.wunderground.com/api/' + key + 
  '/forecast/q/NY/New_York.json'

r = requests.get(ApiUrl) 
forecast = r.json() 
print forecast['forecast']['txt_forecast']['forecastday'][0]
  ['fcttext'] 

Replace this with your API key.

Get the New York City forecast from Weather Underground (replace with your own state and city).

Take the text of the JSON response and parse it into a Python dictionary object.

“Reach into” the data hierarchy to get today’s forecast text.

So you now have a Python script that will output the text forecast whenever you need it. But how can your Raspberry Pi determine whether or not it’s supposed to rain today? On one hand, you could parse the forecast text to search for words like “rain,” “drizzle,” “thunderstorms,” “showers,” and so on, but there’s actually a better way. One of the fields in the data from Weather Underground API is labeled pop, which stands for probability of precipitation. With values ranging from 0 to 100%, it will give you a sense of how likely rain or snow will be.

For a rain forecast indicator, let’s say that any probability of precipitation value over 30% is a day that we want to have an umbrella handy:

1. Connect an LED to pin 25, as you did in Figure 6-5.

2. Create a new file called umbrella-indicator.py and use the code in Example 10-3. Don’t forget to put in your own API key and the location in the Weather Underground API URL.

3. Run the script as root with the command sudo python umbrella-indicator.py.

Example 10-3. Source code for umbrella-indicator.py

import requests
import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
GPIO.setup(25, GPIO.OUT)

key = 'YOUR KEY HERE' 
ApiUrl = 'http://api.wunderground.com/api/' + key + 
  '/forecast/q/NY/New_York.json'

while True:
   r = requests.get(ApiUrl)
   forecast = r.json()
   popValue = forecast['forecast']['txt_forecast']['forecastday']
     [0]['pop'] 
   popValue = int(popValue) 

   if popValue >= 30: 
      GPIO.output(25, GPIO.HIGH)
   else: 
      GPIO.output(25, GPIO.LOW)

   time.sleep(180) # 3 minutes 

As before, change this to your API key.

Get today’s probability of precipitation and store it in popValue.

Convert popValue from a string into an integer so that we can evaluate it as a number.

If the value is greater than 30, then turn the LED on.

Otherwise, turn the LED off.

Wait three minutes before checking again so that the script stays within the API limit of 500 requests per day.

Press Ctrl-C to quit the program when you’re done.

The Weather Underground API is one of a plethora of different APIs that you can experiment with. Table 10-2 lists a few other sites and services that have APIs.

Table 10-2. Popular application programming interfaces

Site	API Reference URL
Facebook	https://developers.facebook.com
Flickr	http://www.flickr.com/services/api
Foursquare	https://developer.foursquare.com
Reddit	https://www.reddit.com/dev/api
Twilio	http://www.twilio.com
Twitter	https://dev.twitter.com
YouTube	https://developers.google.com/youtube

Serving Pi (Be a Web Server)

Not only can you use the Raspberry Pi to get data from servers via the Internet, but your Pi can also act as a server itself. There are many different web servers that you can install on the Raspberry Pi. Traditional web servers, like Apache or lighttpd, serve the files from your board to clients. Most of the time, servers like these are sending HTML files and images to make web pages, but they can also serve sound, video, executable programs, and much more.

However, there’s a new breed of tools that extend programming languages like Python, Ruby, and JavaScript to create web servers that dynamically generate the HTML when they receive HTTP requests from a web browser. This is a great way to trigger physical events, store data, or check the value of a sensor remotely via a web browser. You can even create your own JSON API for an electronics project!

$ sudo pip install flask

from flask import Flask
app = Flask(__name__) 

@app.route("/") 
def hello():
   return "Hello World!" 

if __name__ == "__main__": 
   app.run(host='0.0.0.0', port=80, debug=True) 

$ sudo python hello-flask.py
 * Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger pin code: 238-627-191

10.0.1.100 - - [19/Nov/2012 00:31:31] "GET / HTTP/1.1" 200 -
10.0.1.100 - - [19/Nov/2012 00:31:31] "GET /favicon.ico HTTP/
  1.1" 404 -

from flask import Flask, render_template
import datetime
app = Flask(__name__)

@app.route("/")
def hello():
   now = datetime.datetime.now() 
   timeString = now.strftime("%Y-%m-%d %H:%M") 
   templateData = {
      'title' : 'HELLO!',
      'time': timeString
      } 
   return render_template('main.html', **templateData) 

if __name__ == "__main__":
   app.run(host='0.0.0.0', port=80, debug=True)

<!DOCTYPE html>
   <head>
      <title>{{ title }}</title> 
   </head>

   <body>
      <h1>Hello, World!</h1>
      <h2>The date and time on the server is: {{ time }}</h2> 
   </body>
</html>

Connecting the Web to the Real World

You can use Flask with other Python libraries to bring additional functionality to your site. For example, with the RPi.GPIO Python module (see Chapter 7), you can create a website that interfaces with the physical world. To try it out, hook up three buttons or switches to pins 23, 24, and 25 in the same way as the Simple Soundboard project in Figure 7-2.

The following code expands the functionality of hello-template.py, so copy it to a new file called hello-gpio.py. Add the RPi.GPIO module and a new route for reading the buttons, as we’ve done in Example 10-7. The new route will take a variable from the requested URL and use that to determine which pin to read.

You’ll also need to create a new template called pin.html (it’s not very different from main.html, so you may want to copy main.html to pin.html and make the appropriate changes, as in Example 10-8):

from flask import Flask, render_template
import datetime
import RPi.GPIO as GPIO
app = Flask(__name__)

GPIO.setmode(GPIO.BCM)

@app.route("/")
def hello():
   now = datetime.datetime.now()
   timeString = now.strftime("%Y-%m-%d %H:%M")
   templateData = {
      'title' : 'HELLO!',
      'time': timeString
      }
   return render_template('main.html', **templateData)

@app.route("/readPin/<pin>") 
def readPin(pin):
   try: 
      GPIO.setup(int(pin), GPIO.IN) 
      if GPIO.input(int(pin)) == True: 
         response = "Pin number " + pin + " is high!"
      else: 
         response = "Pin number " + pin + " is low!"
   except: 
      response = "There was an error reading pin " + pin + "."

   templateData = {
      'title' : 'Status of Pin' + pin,
      'response' : response
      }

   return render_template('pin.html', **templateData)

if __name__ == "__main__":
   app.run(host='0.0.0.0', port=80, debug=True)

Add a dynamic route with pin number as a variable.

If the code indented below raises an exception, run the code in the except block.

Take the pin number from the URL, convert it into an integer, and set it as an input.

If the pin is high, set the response text to say that it’s high.

Otherwise, set the response text to say that it’s low.

If there was an error reading the pin, set the response to indicate that.

<!DOCTYPE html>
   <head>
      <title>{{ title }}</title> 
   </head>

   <body>
      <h1>Pin Status</h1>
      <h2>{{ response }}</h2> 
   </body>
</html>

Insert the title provided from hello-gpio.py into the page’s title.

Place the response from hello-gpio.py on the page inside HTML heading tags.

With this script running, when you point your web browser to your Raspberry Pi’s IP address, you should see the standard “Hello World!” page we created before. But add /readPin/24 to the end of the URL so that it looks something like http://10.0.1.103/readPin/24. A page should display showing that the pin is being read as low. Now hold down the button connected to pin 24 and refresh the page; it should now show up as high!

Try the other buttons as well by changing the URL. The great part about this code is that we only had to write the function to read the pin once and create the HTML page once, but it’s almost as though there are separate web pages for each of the pins!

Project: WebLamp

In Chapter 6, we showed you how to use Raspberry Pi as a simple AC outlet timer in “Project: Cron Lamp Timer”. Now that you know how to use Python and Flask, you can now control the state of a lamp over the Web. This basic project is simply a starting point for creating Internet-connected devices with the Raspberry Pi.

And just like how the previous Flask example showed how you can have the same code work on multiple pins, you’ll set up this project so that if you want to control more devices in the future, it’s easy to add:

1. The hardware setup for this project is exactly the same as the “Project: Cron Lamp Timer”, so all the parts you need are listed there.

2. Connect the PowerSwitch Tail II relay to pin 25, just as you did in the Cron Lamp Timer project.

3. If you have another PowerSwitch Tail II relay, connect it to pin 24 to control a second AC device. Otherwise, just connect an LED to pin 24. We’re simply using it to demonstrate how multiple devices can be controlled with the same code.

4. Create a new directory in your home directory called WebLamp.

5. In WebLamp, create a file called weblamp.py and put in the code from Example 10-9.

6. Create a new directory within WebLamp called templates.

7. Inside templates, create the file main.html. The source code of this file can be found in Example 10-10.

In terminal, navigate to the WebLamp directory and start the server (be sure to use Ctrl-C to kill any other Flask server you have running first):

pi@raspberrypi ~/WebLamp $ sudo python weblamp.py

Open your mobile phone’s web browser and enter your Raspberry Pi’s IP address in the address bar, as shown in Figure 10-2.

Figure 10-2. The device interface, as viewed through a mobile browser

import RPi.GPIO as GPIO
from flask import Flask, render_template, request
app = Flask(__name__)

GPIO.setmode(GPIO.BCM)

pins = {
   24 : {'name' : 'coffee maker', 'state' : GPIO.LOW},
   25 : {'name' : 'lamp', 'state' : GPIO.LOW}
   } 

for pin in pins: 
   GPIO.setup(pin, GPIO.OUT)
   GPIO.output(pin, GPIO.LOW)

@app.route("/")
def main():
   for pin in pins:
      pins[pin]['state'] = GPIO.input(pin) 
   templateData = {
      'pins' : pins 
      }
   return render_template('main.html', **templateData) 

@app.route("/<changePin>/<action>") 
def action(changePin, action):
   changePin = int(changePin) 
   deviceName = pins[changePin]['name'] 
   if action == "on": 
      GPIO.output(changePin, GPIO.HIGH) 
      message = "Turned " + deviceName + " on." 
   if action == "off":
      GPIO.output(changePin, GPIO.LOW)
      message = "Turned " + deviceName + " off."
   if action == "toggle":
      GPIO.output(changePin, not GPIO.input(changePin)) 
      message = "Toggled " + deviceName + "."

   for pin in pins:
      pins[pin]['state'] = GPIO.input(pin) 

   templateData = {
      'message' : message,
      'pins' : pins
   } 

   return render_template('main.html', **templateData)

if __name__ == "__main__":
   app.run(host='0.0.0.0', port=80, debug=True)

Create a dictionary called pins to store the pin number, name, and pin state.

Set each pin as an output and make it low.

For each pin, read the pin state and store it in the pins dictionary.

Put the pins dictionary into the template data dictionary.

Pass the template data into the template main.html and return it to the user.

The function below is executed when someone requests a URL with the pin number and action in it.

Convert the pin from the URL into an integer.

Get the device name for the pin being changed.

If the action part of the URL is “on,” execute the code indented below.

Set the pin high.

Save the status message to be passed into the template.

Read the pin and set it to whatever it isn’t (i.e., toggle it).

For each pin, read the pin state and store it in the pins dictionary.

Along with the pins dictionary, put the message into the template data dictionary.

<!DOCTYPE html>
<head>
   <title>Current Status</title>
</head>

<body>
   <h1>Device Listing and Status</h1>

   {% for pin in pins %} 
   <p>The {{ pins[pin].name }} 
   {% if pins[pin].state == true %} 
      is currently on (<a href="/{{pin}}/off">turn off</a>)
   {% else %} 
      is currently off (<a href="/{{pin}}/on">turn on</a>)
   {% endif %}
   </p>
   {% endfor %}

   {% if message %} 
   <h2>{{ message }}</h2>
   {% endif %}

</body>
</html>

Run through each pin in the pins dictionary.

Print the name of the pin.

If the pin is high, print that the device is on and link to URL to turn it off.

Otherwise, print that the device is off and link to the URL to turn it on.

If a message was passed into the template, print it.

The best part about writing the code in this way is that you can very easily add as many devices that the hardware will support. Simply add the information about the device to the pins dictionary. When you restart the server, the new device will appear in the status list and its control URLs will work automatically.

There’s another great feature built in: if you want to be able to flip the switch on a device with a single tap on your phone, you can create a bookmark to the address http://ipaddress/pin/toggle. That URL will check the pin’s current state and switch it.

Going Further

Requests

The homepage for Requests includes very comprehensive documentation complete with easy-to-understand examples.

Flask

There’s a lot more to Flask that we didn’t cover. The official site outlines Flask’s full feature set.

Flask Extensions

Flask extensions make it easy to add functionality to your site.