Let's now explore a couple of examples on HTTP requests. I will use the requests library for these examples, which you can install with pip. We're going to perform HTTP requests against the httpbin.org API, which, interestingly, was developed by Kenneth Reitz, the creator of the requests library itself. This library is amongst the most widely adopted all over the world:

import requests

urls = {
    'get': 'https://httpbin.org/get?title=learn+python+programming',
    'headers': 'https://httpbin.org/headers',
    'ip': 'https://httpbin.org/ip',
    'now': 'https://now.httpbin.org/',
    'user-agent': 'https://httpbin.org/user-agent',
    'UUID': 'https://httpbin.org/uuid',
}

def get_content(title, url):
    resp = requests.get(url)
    print(f'Response for {title}')
    print(resp.json())

for title, url in urls.items():
    get_content(title, url)
    print('-' * 40)

The preceding snippet should be simple to understand. I declare a dictionary of URLs against which I want to perform requests. I have encapsulated the code that performs the request into a tiny function: get_content. As you can see, very simply, we perform a GET request (by using requests.get), and we print the title and the JSON decoded version of the body of the response. Let me spend a word about this last bit.

When we perform a request to a website, or API, we get back a response object, which is, very simply, what was returned by the server we performed the request against. The body of all responses from httpbin.org happens to be JSON encoded, so instead of getting the body as it is (by getting resp.text) and manually decoding it, calling json.loads on it, we simply combine the two by leveraging the json method on the response object. There are plenty of reasons why the requests package has become so widely adopted, and one of them is definitely its ease of use.

Now, when you perform a request in your application, you will want to have a much more robust approach in dealing with errors and so on, but for this chapter, a simple example will do. Don't worry, I will give you a more comprehensive introduction to HTTP requests.

Going back to our code, in the end, we run a for loop and get all the URLs. When you run it, you will see the result of each call printed on your console, like this (prettified and trimmed for brevity):

$ python reqs.py
Response for get
{
  "args": {
    "title": "learn python programming"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.19.0"
  },
  "origin": "82.47.175.158",
  "url": "https://httpbin.org/get?title=learn+python+programming"
}
... rest of the output omitted ...

Notice that you might get a slightly different output in terms of version numbers and IPs, which is fine. Now, GET is only one of the HTTP verbs, and it is definitely the most commonly used. The second one is the ubiquitous POST, which is the type of request you make when you need to send data to the server. Every time you submit a form on the web, you're basically making a POST request. So, let's try to make one programmatically:

# io_examples/reqs_post.py
import requests

url = 'https://httpbin.org/post'
data = dict(title='Learn Python Programming')

resp = requests.post(url, data=data)
print('Response for POST')
print(resp.json())

The previous code is very similar to the one we saw before, only this time we don't call get, but post, and because we want to send some data, we specify that in the call. The requests library offers much, much more than this, and it has been praised by the community for the beautiful API it exposes. It is a project that I encourage you to check out and explore, as you will end up using it all the time, anyway.

Running the previous script (and applying some prettifying magic to the output) yields the following:

$ python reqs_post.py
Response for POST
{ 'args': {},
  'data': '',
  'files': {},
  'form': {'title': 'Learn Python Programming'},
  'headers': { 'Accept': '*/*',
               'Accept-Encoding': 'gzip, deflate',
               'Connection': 'close',
               'Content-Length': '30',
               'Content-Type': 'application/x-www-form-urlencoded',
               'Host': 'httpbin.org',
               'User-Agent': 'python-requests/2.7.0 CPython/3.7.0b2 '
                             'Darwin/17.4.0'},
  'json': None,
  'origin': '82.45.123.178',
  'url': 'https://httpbin.org/post'}

Notice how the headers are now different, and we find the data we sent in the form key/value pair of the response body.

I hope these short examples are enough to get you started, especially with requests. The web changes every day, so it's worth learning the basics and then brush up every now and then.

Let's now move on to the last topic of this chapter: persisting data on disk in different formats.