5
Python on the Web

WHAT YOU WILL LEARN IN THIS CHAPTER:    

  • Understanding how Python works on the web
  • Creating a web app with Python
  • Connecting a web app to a database
  • Creating an API
  • Parsing and manipulating data in a web app

WROX.COM DOWNLOADS FOR THIS CHAPTER

For this chapter the wrox.com code downloads are found at www.wrox.com/go/pythonprojects on the Download Code tab. The code is in the Chapter 5 download, called Chapter5.zip, and individually named according to the names throughout the chapter.

Up to this point you’ve been using Python locally to investigate and parse data on your local machine. But what about when you want to use Python for remote data manipulation, or across “the wire”?

Python is a very powerful language, and it’s not just for doing system administration tasks or file system tasks locally. You can use Python across networks to handle tasks using REST and XMLRPC. This chapter looks into both of these, but the focus is on the most popular method of using Python remotely: HTTP using REST.

In this chapter you use some of the more popular web technologies with Python, including HTTP and REST, to create a web app that links to a database. You use technologies such as Flask and SQLite to complete the web app, and finally, you create an application programming interface (API) and learn to parse and manipulate data in the web app.

REST stands for Representational State Transfer. It’s a fancy name for the technology and architecture style of a certain web service. This service doesn’t rely on the implementation of components or protocol syntax; instead, it worries about how to interact with data (sending it back and forth within the constraints placed upon it). We explain RESTful architecture in the section on building an API with Flask. For now, just know that REST is a thing that makes up the architecture of web services.

Python on the Web

As you may know, the web is made up of a few technologies, not just one. Figure 5.1 shows a high-level overview of the structure of a modern web app.

images

Figure 5.1 Note the two sides of a web app: client-side and server-side. Server-side data is served via a web server such as Apache or Nginx.

As you can see, the front end of a web app consists of a browser that handles the HTML, CSS, and JavaScript. The middle layer, in this case, is Python, but in general this is usually some scripting language such as Ruby, Perl, or PHP, or JavaScript. The back end will house your database (SQLite) and web server, or HTTP Daemon (commonly, Nginx or Apache). Although it is beyond the scope of this book to explain the fundamentals of the back end and front end, you will learn how to set up a very simple server.

Let’s move on and start explaining how each of these pieces work together.

Parts of a Web Application

You can serve data across networks in a few ways. Python is a language that can handle almost all of those ways. We briefly touch on the most common uses of Python as a server-side language, and then we use the most modern and most common way of using Python on the web: building a web application.

The basic structure of any network request is shown in Figure 5.2. What happens between “Server Receives Request” and “Server Returns Data to Requestor”? That is where Python lives.

images

Figure 5.2 Structure of a network request

Because browsers can’t ship with every interpreter ever created—and we certainly don’t want them to come with compilers—we needed to find a way to bridge static pages and dynamic data so that we could have interactive, dynamic web pages. We know that we have a web server that is serving up files that make up our web app, but how does the server know what files we’re using? How can the server tell that the app is written in Python and not, say, Ruby?

Welcome WSGI! WSGI, or Web Server Gateway Interface, is how your web server (like Apache) knows about and can run Python files. It is beyond the scope of this book to delve into the intricacies of WSGI; just know that most modern web servers support it, and if you want to run Python on a server, that server needs to have WSGI available. If you’re interested in a deeper understanding of server operations and what is known as “DevOps,” it is recommended that you research the more lower-level technologies, such as WSGI. For our purposes, however, we’ll be using a framework that has already taken care of that piece of server architecture for us and uses WSGI. You will see this term referenced throughout when discussing Python on the web, so hopefully you’ll remember it and understand what part it plays in the grand scheme of things.

The Client-Server Relationship

What is a client? In web development, a client usually refers to a web browser, which sends out requests to your web server for files. However, you will see other uses of the term “client” to mean another server that is requesting data from a second server. For the purposes of this chapter, a client is basically a browser—in other words, the part of the web app that wants to display the data that the app is returning. This is also the part of the web app that the user will be interacting with. If you’ve ever heard the term “client-side JavaScript” and never really understood it, this should clear up that confusion: Client-side JavaScript is JavaScript that is only on the client side and performs actions directly in the client (a web browser), and not on the server. This includes, for example, changing colors or styles on a webpage when certain events are fired (i.e., a button-click changes the color of a background). These functions never actually talk to a server and can be used locally, if needed (without an Internet connection).

So, what is server-side JavaScript, then? As you probably guessed, it’s JavaScript that is executed on the server, usually by an event from the client side, such as a GET request or client-side JavaScript. We won’t be dealing with any server-side JavaScript in this book because we’re using Python, which is our server-side language of choice. This is where our logic will live. This is the part of the app that takes the actions from the client and makes magic happen.

Middleware and MVC

Middleware is a fairly new term in web development. It refers to the part of the technology stack that will take in data from the front end (the client), manipulate it, pass it into or out of the database (or other service that may be running), and then send it back to the front end. Basically, the logic of your system/app should live in your middle layer, your data should live in your data layer, and your styles should live in the front end. It is never ideal to have your front end (JavaScript) doing much data logic, which can be better handled in your middle layer.

All of these pieces actually have a nice name—it is called model-view-controller (MVC). Most modern frameworks use some sort of MVC architecture. The benefit of MVC is that your client doesn’t have to deal with the logic of your app, and your logic doesn’t have to deal with your data models. You can set your data models and forget 'em! (sorta).

Say you’re working on a large project and you have web designers writing your HTML and CSS and some of the JavaScript that makes your web app look slick and shiny. Then you have very smart mathematicians working on your data layer, because you need precise numbers to be calculated from data inputted from your slick, shiny client. What happens if you have your logic code in your client-side files? Let’s say you put your calculations in the templates that the web designers were working on. Furthermore, let’s say that your web designers wanted to manipulate something and they thought your calculation was the problem. This isn’t ideal, is it? MVC somewhat solves this problem by separating out your data layer (the data model that is storing your precise calculations), from the controllers you’re writing to do your precise calculations, with the view that your web designers are working on to make a shiny, slick web app.

So, what does all this mean in the Python world? Well, you’re going to come across the term “MVC” and you should understand the overview of the architecture. We use a framework in this chapter that utilizes a sort of MVC architecture, so you should now have a better understanding of why we’ve made most of the decisions we’ve made, moving forward.

HTTP Methods and Headers

HTTP stands for HyperText Transfer Protocol. This is the protocol that is used to pass data around on the web, usually via a web browser. When a client makes a request, it uses this HTTP protocol. This might look familiar to anyone who’s done any web programming with HyperText Markup Language (HTML). HTTP is made to pass HTML.

HTTP methods are the verbs of the web. The two most basic methods are GET and POST. These do what you might expect. When you send the web server a GET request, you are requesting to get data from the server. This request, or GET, is used every time you load a web page into your browser. So, when you go to twitter.com, your browser is sending a GET request to the Twitter servers and asking for information.

What about when you type out a tweet and hit the Tweet button? When you are sending data to the server, you are using a POST method call. This indicates to the server that you are going to attempt to post data to the server. (We lightly touch on some other method calls as you go through the process of building your app.)

So, how does the web server know what method is being sent? It does this via the HTTP headers (see Figure 5.3).

images

Figure 5.3 The Chrome Developer Tools, illustrating the headers of a server request for a file

As you can see in Figure 5.3, we have gone to http://www.python.org and in doing so, have sent a GET request to the server for a JavaScript file that is named iotbs2-core.js. The server responded with a status code of “200 OK,” which simply means that the file was available to be served and was served. Other HTTP Status Codes exist: the code 404 is the most widely known—it means “Not Found,” meaning the file was not found on the server, so it could not be served.

Headers contain information for the server, to know what you are requesting from the server. This helps the HTTP, or web, server to respond properly to requests that are made from the Internet. Headers also contain metadata, such as what browser was being used to make the request. As you can see in Figure 5.3, the request was made using Chrome on a Macintosh running OS X 10.8.5.

If you’re going to be developing for the web and you don’t have a dedicated front-end team to check your API’s implementation, you’re going to have to do your own debugging. It’s also helpful to know how your API will be accessed and to be able to replicate that for testing and debugging purposes. To do this, you will be using the Chrome Developer Tools (or DevTools, for short).

images

Figure 5.4 The DevTools Network tab

images

Figure 5.5 FigurePreview of the selected file

images

Figure 5.6 Headers for the selected file

What Is an API?

API stands for application programming interface. An API is simply the approved way for others to interact with an outside application, without actually having access to the database itself. Take Twitter, for example. How do the many Twitter clients out there get the Twitter data? Does Twitter just let anyone onto its systems? No, not really. Twitter uses an API to allow other developers to utilize its data. The modern use of an API is a RESTful API, meaning that you use URLs to talk to the remote application’s API, and the remote application returns data depending on the data you’ve sent to the specified URL.

You can find many tutorials online that show you how to use certain libraries in Python to access an API and pull data. You will be going a step farther than that—you will create your own API to serve data. You will then use that API to code up your client-side files to pull in the correct data. But first, let’s look at the data that most APIs return: JSON.

In the following Try It Out, you learn how to access a third-party API to get data. For this example, you use the USDA’s API for Farmer’s Markets in a given area. You can find plenty of other governmental APIs at www.data.gov. You can find the documentation for this example at http://search.ams.usda.gov/farmersmarkets/v1/svcdesc.html.

So, now the question is: How does that work? How does the Requests library do that? That’s for our next section, “Web Programming with Python.” Though we won’t be making our own requests-like library, we will look at the underlying logic and functionality that Python provides in order to make such a library.

Web Programming with Python

In the preceding section, you took a look at the Requests library. In this section you’re going to see a little bit of the technologies Requests employs under the hood. Then, using the Flask web framework (http://flask.pocoo.org), you’ll take the lending library, give it a web interface, and make it interactive via a browser. But first, you should understand just how things are working before you put big bows on them.

Using the Python HTTP Modules

Python is incredibly powerful, versatile, and easy to use. Part of the reason for this is all the built-in functionality that comes with the base Python install. When you install Python, you also get a plethora of modules that you can use. Included in these modules are the http modules—http
.server, http.client—and a few others. In this section you work with the http.server module to see just how easy it is to spin up a quick HTTP server so that you can quickly begin serving web pages for debugging/testing purposes. This section also illustrates just how some of the more popular third-party libraries use Python’s built-in modules to create incredibly powerful tools for the Python developer.

Let’s take a look at the http.server module and see just how quickly you can get an HTTP server up and running locally.

Creating an HTTP Server

In this Try It Out, you set up, in just a few lines of code, an HTTP server that will run locally and serve up pages that you’ll create and serve out of your local directory. After this exercise, you should have a good idea of just how some of the more popular frameworks and third-party libraries implement Python’s built-in modules to harness incredible power to create easy-to-use tools.

Exploring the Flask Framework

Now, you take a closer look at the Flask framework. It is a third-party library written in Python that will help you to do everything from creating your own APIs to creating a full web-based application. Flask supports templating, using the Jinja template system, and therefore you can use it as an all-in-one web framework. You use this functionality for the next part of this chapter.

Before you begin, you’ll need to install Flask (pip install flask). You’ll be using SQLite3 for your database. Luckily, SQLite3 comes with Python 3. SQLite is a smaller database that is perfect for this simple example app. If you’re doing a larger app, we recommend you take the time to research larger databases such as PostgreSQL, SQLAlchemy, or many of the others available, including the NoSQL databases such as Redis. For the purposes of this chapter, however, you’ll use SQLite, which is a very easy, small, and lightweight database.

Once you have Flask installed, you’ll set up a very simple app, which you will build on to complete the larger app.

To start, you want to set up the directory structure where each part of your project will live. You can set up projects in many different ways, and each tutorial shows you a different way, but the basic idea is the same. Because you’ll be working in an MVC-like environment, you’re going to want to separate out each piece of the project. In the following Try It Out, once you unzip the lendy.zip file, you should have a directory structure similar to this:

  • lendy (the silly name of your lending app)
    • static
    • templates

This structure is similar to how most projects start out. You want to separate each piece of the project. The static directory will hold static files, such as images and scripts. The templates directory is where the templates will live.

Let’s look at each piece of a typical MVC architecture:

  • Model: This is where the data models live, or the tables in the database. This is going to be the table for your users (not secure) and the items that you’ll be lending.
  • View: This is what the users will see—what you show to them in a browser; or how they interact with the data, and where the data will go, once the next step does its job.
  • Controller: This is the part of the app that “controls” the data flow. Users interact with this part. So, a user may send form data to the back end, which is picked up via the controller part of the project. This part interacts with the model and can initiate either an update, creation, or even deletion. This is also the “heaviest” part of the app, in that this part receives data from the model layer and returns it to the view. This is, basically, the brain of your web app.

Let’s see how all of these pieces fit together by creating the basics for your web app.

Creating Data Models in Flask

In this example you will be use the lendydb database that you created in Chapter 3, in the section “Creating the LendyDB SQL Database.” You will use two of the data entities: a member and an item. Members will be able to log in and view the inventory of items and add items to the inventory. You will be using a text editor and saving files in this exercise. You will not be using any security measures, for the sake of time. However, you should be very familiar with the most common security practices before setting up public web interfaces.

The data model for this application consists of the two files from Chapter 3: lendydata.py, which is the API, and lendy.db, which is the actual database. You need to copy both of these into your lendy folder.

Creating Core Flask Files

Now it’s time to really get to the meat of the app—everything that will give us functionality and allow us to have a working web app. Let’s create the Python files for the app.

More on Python and the Web

Creating web apps with Python is incredibly easy. You can do all the heavy lifting yourself, or you can try one of the many open source frameworks out there to help you with it, or help with just a few parts. You can build websites in two main ways with Python: static site generators and full-on web frameworks. This section briefly describes both methods and includes a list of some of the more popular generators and frameworks.

Static Site Generators

Static site generators are usually used for things like blogs and other documentation—where you may make a new page at a time and want to serve out that page but the content on the page won’t be changing once it’s published. Some of the more popular static site generators include, but are not limited to:

  • Pelican: Probably the most popular and well-known static site generator in Python. There is also lots of community support, including plug-ins and themes for Pelican.
  • Hyde: A little larger than Pelican, with a bit more of a learning curve, but quite robust.
  • Nikola: Fully featured, lots of community involvement and support. Also supports customization, including themes and plug-ins.
  • Mynt: Used by www.pyladies.com and a few other sites. Mynt lauds itself as having the features of a content management system (CMS) without the rigid implementation.

Web Frameworks

Web frameworks are all-in-one systems that give you the power to create APIs, web apps, and even a comprehensive CMS. Here are a few we recommend:

  • Flask: Flask is quite versatile, from creating simple APIs that others will access to creating full web apps.
  • Django: Django is quite popular and very robust. It even includes an admin interface that allows users to put entries in the database with an easy-to-navigate user interface that is highly customizable.
  • Bottle: Bottle is smaller than Flask and simply gives you just what you need to create a website, with very little overhead and limited functionality. It’s perfect for smaller websites and pages.
  • Pyramid: Pyramid is similar to Flask in that it can be used for small projects, but you can also use it for larger projects, and it can be scaled up as needed.

Of course, many more web frameworks are available. We recommend finding one that suits you and your project, or just play around with a few to find one you feel more comfortable with.

Using Python Across the Wire

You can do many different tasks across a network connection. Whether that connection is the public Internet, a local area network, or a private network, you can process data, serve web pages, and even run Python scripts remotely. This section introduces a few of the simple, different ways you can run Python across a wire. None of the examples are production-ready code, but this should be a good way to get you familiar with the different powers Python can provide.

XML-RPC

XML-Remote Procedure Call (XML-RPC) is an older technology that is still used in a few legacy systems. This is how data was once processed across the Internet, using XML. We now use JSON to pass data back and forth; however, some systems still use XML and require remote procedural calls. Because of this Python has a built-in XML-RPC module. Let’s make one and watch it in action.

In this Try It Out, you set up, in just a few lines of code, an XML-RPC server that will run locally in one Terminal and serve up a simple Python script, which you create. You then open another Terminal window and run the code remotely.

Many people ask just what you could use such functionality for in the real world. XML processing was once the way we processed data across the wire. Some organizations still use this method. However, the power that we wanted to illustrate was that you can even set up Python to run remotely. So, if you had a large data file and wanted to process it remotely, you could set up code on a remote server and then call that code from another machine. Next, you’ll look at some other examples that can also accomplish these tasks.

Socket Servers

If you’ve ever heard anyone talk about “Websockets” or “streaming data,” they’re usually talking about TCP and socket servers. These servers utilize TCP, or Transmission Control Protocol. You may have heard of TCP/IP, which this is the same thing. Python, of course, has some built-in libraries that can enable you to create TCP sockets to send and receive data between two points using the Internet Protocol.

In this Try It Out you get a bit more involved with some Python, and do some things in a more “pythonic” way. You’re going to again set up a server and a client, so you’ll need two Terminal windows running to complete this task. However, you’re also going to create a class to handle your TCP requests to illustrate the use of classes in Python.

We’ve only shown you the very basics of what Python sockets can do. Our job here is to familiarize you with all the tools that are available in the Python ecosphere. There may be times when a direct data connection is needed, for updating data feeds in real time—this is where the power of sockets and streaming data comes in very handy. Luckily for you, Python makes this fairly easy. We, of course, recommend that if you’re going to be creating network connections on a lower level, you understand the security risks and cautions that you will need to be aware of before undertaking such a task.

More Networking Fun in Python

You may be interested in delving a bit deeper into networking with Python. If so, here is a short list of some of the more popular networking libraries available for download:

  • Twisted (http://twistedmatrix.com): Twisted is very large, very powerful, and full of networking goodness. However, as of this writing it is not fully functional with Python 3. If you’re interested in doing some event-driven networking, Twisted is for you. Support for SMTP, POP3, IMAP, SSHv2, and DNS is included. So if you’ve always wanted to make your own e-mail server, you and Twisted may be a match made in heaven. If you’re interested in setting up your own SSL server—get Twisted!
  • Tornado (http://www.tornadoweb.org): Tornado is lauded as a web framework and asynchronous networking library. Mainly made for larger applications that may need long-lived connections, Tornado uses non-blocking I/O and is perfect for Websockets and long polling. We put Tornado in the “networking” section rather than the “web frameworks” section because it’s more focused on the networking side of web frameworks than on creating templates and making things pretty.
  • gevent (http://www.gevent.org): According to gevent’s own website, gevent is “a coroutine-based Python networking library….that provides a high-level synchronous API…” When you need to do some crazy coroutines across your network, you may want to take a gander at gevent. Currently, gevent is only for Python 2.

You should now have a good idea of the power and ease with which Python can be used to send and receive data across the wire. If you’re interested in any sort of data passing using Python, this chapter should have given you a nice jumping-off point to go explore more and hopefully create interesting things with your discoveries.

Python’s community is full of helpful people, so if you happen to find a framework or library that you particularly like, join the mailing lists, the IRC channels, and any other conversations you can find. Contribute to those projects, and help the communities grow larger and stronger. Participation in technologies that you find interesting and/or helpful not only helps the projects and organization that is offering the technology, but it helps all developers who may need to use or want to learn that technology.

Summary

You started off this chapter by learning about how Python works on the web. The front end of a web app consists of a browser that handles the HTML, CSS, and JavaScript. The middle layer, in this case, is Python. The back end houses your database (SQLite) and web server. You also learned about APIs (application programming interfaces), which is the approved way for others to interact with an outside application, without actually having access to the database itself. Next, you practiced using a third-party API and the Requests library. Along with that, you explored the technologies Requests employs under the hood. Then, using the Flask web framework, you took the lending library, gave it a web interface, and made it interactive via a browser. Finally, you learned a few of the various easy ways you can run Python across a wire.

Exercises

  1. Consider our code from earlier in this chapter:

      >>>for result in results['results']:
  ...		 id = result['id']
  ...		 print(id)
  ...		 print (result['id'])
  ...		 details = requests.get(market + id).json()
  ...		 print (details)
  ...		 print (details['marketdetails'])
  ...	print (details['marketdetails']['GoogleLink'])

    Using what you know about Python, can you figure out a way to create a list comprehension that will do the same thing as the preceding code? Remember that list comprehensions are constructed like this:

      [expression for item in list if conditional]

  2. Using what you know so far about how to use files in Python, can you save the output of your call to the USDA's API to a file on your machine, to parse later? (Is saving it as a .txt file fine?)

  3. Can you find the docs for Flask that would help us to break our app into smaller, modularized files with our endpoints/views in a separate file, rather than having one big Python file with everything in it? (Hint: It is one concept/feature that Flask offers.)

  4. What other HTTP methods can you find? Can you find ways to use them in a Flask app?

  5. By reading the Requests docs, can you find the method call needed to output the HTML of a website by passing the URL to a requests method?

arrow2  WHAT YOU LEARNED IN THIS CHAPTER

TOPIC KEY CONCEPTS
Where Python fits on the web Python is the language that the server uses to manipulate the actual data that is being passed back and forth between machines on the web.
HTTP and HTTP methods The basic “language” of how machines communicate over the web. GET and POST methods and their purposes.
APIs Application programming interface, or how to interface with other servers in order to retrieve or manipulate data. You can produce or consume APIs.
How to create a Flask app The Flask framework is very powerful for creating web apps using Python as the server-side code. This also illustrated how to produce an API.
Templates in Flask Templates are very common in web apps. They are ways to introduce logic and create dynamic content on web pages.
XMLRPC Python has built-in functionality to create XMLRPC servers to pass data over the wire. This is really helpful only to people who will be supporting older/legacy systems, which still use this method of data passing.
Socket servers Python has a very handy built-in library named “socketserver,” which will allow you to create sockets to connect to via other scripts. This is incredibly powerful for processing data over the wire.
SimpleHTTPServer This is a way to make a simple HTTP server to test files locally or to integrate into making a larger framework to use as a debug mode.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset