6. Expanding Your Web Server

In this last chapter of Part II, “Learning to Run,” you expand the web server a little bit with some key new functionality; you learn how to serve up static content such as HTML pages, JavaScript files, cascading style sheets, and even image files. Armed with this knowledge, you turn your focus away from the server and look at programming the client.

In the new Node.js world of programming websites, you’re going to migrate from the traditional model of generating HTML on the server before sending it down to clients and instead have your server serve up only static files or JSON. The web browser can then use AJAX calls along with template libraries to generate pages on the fly as the user navigates around the site. Finally, you look at uploading files to your servers and see some of the tools you can use to help make that task easier.

You begin by taking another look at Node.js streams.

Serving Static Content with Streams

In the asynchronous, nonblocking IO world of Node.js, you previously saw that you can use fs.open and fs.read in a loop to read the contents of a file. However, Node.js provides another, far more elegant mechanism for reading (and even writing) files called streams. They act an awful lot like UNIX pipes—even on Windows—and you briefly saw them in use in the section “Receiving JSON POST Data” in Chapter 4, “Writing Simple Applications,” when you needed to load in the data users sent along with requests.

In the basic usage, you use the on method to add listeners to events. The provided functions are called whenever one of those events is triggered. The readable event is sent whenever a read stream has read something in for you to process. The end event is sent whenever a stream has nothing more to read, and error events are sent whenever something has gone wrong.

Reading a File

As a simple example, write the following code and save it to simple_stream.js:

var fs = require('fs');
var contents;

var rs = fs.createReadStream("simple_stream.js");

rs.on('readable', () => {
    var str;
    var d = rs.read();
    if (d) {
        if (typeof d == 'string') {
            str = d;
        } else if (typeof d == 'object' && d instanceof Buffer) {
            str = d.toString('utf8');
        }
        if (str) {
            if (!contents)
                contents = d;
            else
                contents += str;
        }
    }
});

rs.on('end', () => {
    console.log("read in the file contents: ");
    console.log(contents.toString('utf8'));
});

If you’re looking at the preceding code (it creates an object, adds two listeners, and then does...seemingly nothing) and wondering why it doesn’t just exit before the loading is done, recall in Chapter 3, “Asynchronous Programming,” I said that Node runs in an event loop waiting for things to happen and executing code when something finally does happen.

Well, part of this is knowing when events are pending or going to happen, such as the case in which you have a read stream open and it actually has calls to the file system waiting to finish reading in content. As long as there is something that is expected to happen, Node does not exit until all those events are finished and user code (if there is any) is executed.

The preceding example creates a new read stream given a path with the fs.createReadStream function. It then simply reads itself and prints the contents to the output stream. When it gets a readable event, it calls the read method on the stream and gets back whatever data is currently available. If no data is returned, it just waits until another readable event comes in or an end event is received.

Serving Static Files in a Web Server with Buffers

For this next exercise, write a little web server that serves up static content (an HTML file) using Node buffers. You start with the handle_incoming_request function:

function handle_incoming_request(req, res) {
    if (req.method.toLowerCase() == 'get'
        && req.url.substring(0, 9) == '/content/') {
        serve_static_file(req.url.substring(1), res);
    } else {
        res.writeHead(404, { "Content-Type" : "application/json" });

        var out = { error: "not_found",
                    message: "'" + req.url + "' not found" };
        res.end(JSON.stringify(out) + " ");
    }
}

If an incoming request requests /content/something.html, you try to serve that up by calling the serve_static_file function. The design of the http module in Node.js is sufficiently clever that the ServerResponse object you get to each request on your server is itself actually a stream to which you can write your output! You do this writing by calling the write method on the Stream class:

function serve_static_file(file, res) {
    var rs = fs.createReadStream(file);
    var ct = content_type_for_path(file);
    res.writeHead(200, { "Content-Type" : ct });

    rs.on('readable', () => {
        var d = rs.read();
        if (d) {
            if (typeof d == 'string')
                res.write(d);
            else if (typeof d == 'object' && d instanceof Buffer)
                res.write(d.toString('utf8'));
        }
    });

    rs.on('end', () => {
        res.end();  // we're done!!!
    });
}

function content_type_for_path(file) {
    return "text/html";
}

The rest of the server’s code is as you have seen before:

var http = require('http'),
    fs = require('fs');

var s = http.createServer(handle_incoming_request);
s.listen(8080);

Create a file called test.html in the content/ folder with some simple HTML content in it; then run the server (assuming we saved the code we just wrote into server.js) with

node server.js

And then ask for that test.html file using curl:

curl -i -X GET http://localhost:8080/content/test.html

You should see output similar to the following (depending on what exactly you put in test.html):

HTTP/1.1 200 OK
Date: Mon, 26 Nov 2012 03:13:50 GMT
Connection: keep-alive
Transfer-Encoding: chunked

<html>
<head>
  <title> WooO! </title>
</head>
<body>
  <h1> Hello World! </h1>
</body>
</html>

You are now able to serve up static content!

There is a small problem you might have to deal with: what happens if the write stream cannot accept data as fast as you’re sending it—perhaps the disk can’t write that fast or the network is particularly slow? In these cases, the write method on the streaming object will return false. When you get this, you will need to pause your reading stream and listen for the drain event on the writing stream. Once you get this drain event, then you can resume on the read stream, as follows:

function serve_static_file(file, res) {
    var rs = fs.createReadStream(file);

    var ct = content_type_for_path(file);
    res.writeHead(200, { "Content-Type" : ct });

    rs.on('error''error', (e) => {
        res.writeHead(404, { "Content-Type" : "application/json" });
        var out = { error: "not_found",
                    message: "'" + file + "' not found" };
        res.end(JSON.stringify(out) + " ");
    });

    rs.on('readable', () => {
        var data = rs.read();
        if (!res.write(data)) {
            rs.pause();
        }
    });

    res.on('drain', () => {
        rs.resume();
    });

    rs.on('end', () => {
        res.end();  // we're done!!!
    });
}

With that fixed, we still have two rather serious problems. First, what happens if you ask for /content/blargle.html? In its current form, the script throws an error and terminates, which isn’t what you want. In this case, you want to return a 404 HTTP response code and perhaps even an error message.

To do this, you can listen to the error event on read streams. Add the following few lines to the serve_static_file function:

rs.on('error', (e) => {
        res.writeHead(404, { "Content-Type" : "application/json" });
        var out = { error: "not_found",
                    message: "'" + file + "' not found" };
        res.end(JSON.stringify(out) + " ");
        return;
    }
);

Now, when you get an error (after which no data or end events are called), you update the response header code to 404, set a new Content-Type, and return JSON with the error information that the client can use to report to the user what has happened.

Serving Up More Than Just HTML

The second problem is that you can currently serve up only HTML static content. The content_type_for_path function only ever returns "text/html". We want to be more flexible and serve up other types, which you can accomplish as follows:

function content_type_for_file (file) {
    var ext = path.extname(file);
    switch (ext.toLowerCase()) {
        case '.html': return "text/html";
        case ".js": return "text/javascript";
        case ".css": return 'text/css';
        case '.jpg': case '.jpeg': return 'image/jpeg';
        default: return 'text/plain';
    }
}

Now you can call the curl command with a number of different file types and should get the expected results. For binary files such as JPEG images, you can use the -o flag to curl to tell it to write the output to the specified filename. First, copy a JPEG to your content/ folder, run the server with node server.js, and then add the following:

curl -o abc.jpg http://localhost:8080/content/test.jpg

Chances are, you now have a file called abc.jpg that is exactly what you expect it to be.

Shuffling data from stream (rs in the preceding example) to stream (res) is such a common scenario that the Stream class in Node.js has a convenience method to take care of all this for you: pipe. The serve_static_file function then becomes much simpler:

function serve_static_file(file, res) {
    var rs = fs.createReadStream(file);
    var ct = content_type_for_path(file);
    res.writeHead(200, { "Content-Type" : ct });

    rs.on('error', (e) => {
        res.writeHead(404, { "Content-Type" : "application/json" });
        var out = { error: "not_found",
                    message: "'" + file + "' not found" };
        res.end(JSON.stringify(out) + " ");
        return;
    });

    rs.pipe(res);
}

Assembling Content on the Client: Templates

In more traditional web application models, the client sends an HTTP request to the server, the server gathers all the data and generates the HTML response, and it sends that down as text. While this way of doing things is very reasonable, it has a few key disadvantages:

Image It doesn’t take advantage of any of the computing power available on the average client computer these days. Even the average mobile phone or tablet is many times more powerful than PCs of 10 years ago.

Image It makes your life more difficult when you have multiple types of clients. Some people might access you through a web browser, others through mobile apps, even others through desktop applications or third-party apps.

Image It’s frustrating to have to perform so many different types of things on the server. It would be great if you could have your server focus just on processing, storing, and generating data, letting the client decide how to present the data.

An increasingly common way of doing things, and something I’ve found particularly compelling and fun with Node, is to convert the server back end to serve only JSON, or as absolutely little of anything else as possible. The client app can then choose how to present the returned data to the user. Scripts, style sheets, and even most HTML can be served from file servers or content delivery networks (CDNs).

For web browser applications, you can use client-side templates (see Figure 6.1). In this way of doing things,

1. The client downloads a skeleton HTML page with pointers to JavaScript files, CSS files, and an empty body element from the Node server.

2. One of the referenced JavaScript files is a bootstrapper that does all the work of gathering everything and putting the page together. It is given the name of an HTML template and a server JSON API to call. It downloads the template and then applies the returned JSON data to that template file using a template engine. You will use a template engine called “Mustache” (see the sidebar “Template Engines”).

3. The resulting HTML code is inserted into the body of the page, and all the server has to do is serve up a little bit of JSON.

Image

Figure 6.1 Client-side page generation with templates

It has been in preparation for this that you have been developing the photo album application as a JSON-only server thus far. Although you will add the capability now for it to serve up static files, this is largely for convenience and educational purposes; in a production environment, you would probably move these files to a CDN and update the URLs to point to them as appropriate.

To convert your photo album to this new world of templates, you need to do the following:

1. Generate the HTML skeleton for your pages.

2. Add support to the app for serving static file content.

3. Modify the list of supported URLs to add /pages/ and /templates/ for HTML pages and templates, respectively.

4. Write the templates and JavaScript files that load them.

Let’s go!

The HTML Skeleton Page

Although you might attempt to do all HTML generation on the client, you cannot completely avoid the server doing any at all; you still need the server to send down the initial page skeleton that the client can use to perform the rest of the generation.

For this example, you use a reasonably simple HTML page, which is saved as basic.html, shown in Listing 6.1.

Listing 6.1 The Simple App Page Bootstrapper (basic.html)


<!DOCTYPE html>
<html>
<head>
  <title>Photo Album</title>

  <!-- Meta -->
  <meta http-equiv="content-type" content="text/html; charset=utf-8" />

  <!-- Stylesheets -->
  <link rel="stylesheet" href="http://localhost:8080/content/style.css"
        type="text/css" />

  <!-- javascripts -->
  <script src="http://localhost:8080/content/jquery-1.8.3.min.js"
          type="text/javascript"></script>
  <script src="http://localhost:8080/content/mustache.js"
          type="text/javascript"></script>
  <script src="http://localhost:8080/content/{{PAGE_NAME}}.js"
          type="text/javascript"></script>
</head>
<body></body>

</html>


This file is straightforward; it has a style sheet, and then you include jQuery and Mustache. The last script tag is the most interesting: this file is the page bootstrapper that holds the code that will download the template, get the JSON data from the server for the page, and generate the HTML for the page based on these two things. You can see that it is set up with a parameter {{PAGE_NAME}} to replace with the actual page you will be loading at any given time. (Note that the {{ and }} mustache characters here indicate a variable that will be replaced on the server, not via mustache.js on the client—we just chose them since they’re familiar).)

Serving Static Content

Now you are going to modify the application server’s folder layout a little bit as follows:

+ project_root/
    + contents/    # JS, CSS, and HTML files
    + templates/   # client-side HTML templates
    + albums/      # the albums we've seen already

You’ve seen the albums/ folder before, but now you are going to add the contents/ and templates/ folders to handle additional content you’re going to add.

The updated handle_incoming_request function looks something like the following. The serve_static_file function is identical to the one you just wrote in the preceding section.

function handle_incoming_request(req, res) {
    // parse the query params into an object and get the path
    // without them. (true for 2nd param means parse the params).
    req.parsed_url = url.parse(req.url, true);
    var core_url = req.parsed_url.pathname;

    // test this updated url to see what they're asking for
    if (core_url.substring(0, 9) == '/content/') {
        serve_static_file(core_url.substring(1), res);
    } else if (core_url == '/albums.json') {
        handle_list_albums(req, res);
    } else if (core_url.substr(0, 7) == '/albums'
               && core_url.substr(core_url.length - 5) == '.json') {
        handle_get_album(req, res);
    } else {
        send_failure(res, 404, invalid_resource());
    }
}

For example, in the content/ folder, you start with three files:

Image jquery-1.8.3.min.js—You can download this file directly from jquery.com. Most versions of jQuery from the past couple of years should work just fine.

Image mustache.js—You can download this file from a number of places. In this case, simply use the generic mustache.js from github.com/janl/mustache.js. There are also some server-side versions of mustache as well, which are not what we want for this chapter.

Image style.css—You can just create this as an empty file or start to fill in CSS to modify the pages that you’ll generate for the album app.

Modifying Your URL Scheme

The app server thus far supports only /albums.json and /albums/album_name.json. You can now add support for static content, pages and template files as follows:

/content/some_file.ext
/pages/some_page_name[/optional/junk]
/templates/some_file.html

For content and template files, there are actually real files to download directly from the hard disk. For page requests, however, you always return some variation on the basic.html you saw in the previous section. It has the {{PAGE_NAME}} macro replaced with the name of the page passed in through the URL; for example, /page/home.

The updated handle_incoming_request now looks as follows:

function handle_incoming_request(req, res) {
    // parse the query params into an object and get the path
    // without them. (true for 2nd param means parse the params).
    req.parsed_url = url.parse(req.url, true);
    var core_url = req.parsed_url.pathname;

    // test this fixed url to see what they're asking for
    if (core_url.substring(0, 7) == '/pages/') {
        serve_page(req, res);
    } else if (core_url.substring(0, 11) == '/templates/') {
        serve_static_file("templates/" + core_url.substring(11), res);
    } else if (core_url.substring(0, 9) == '/content/') {
        serve_static_file("content/" + core_url.substring(9), res);
    } else if (core_url == '/albums.json') {
        handle_list_albums(req, res);
    } else if (core_url.substr(0, 7) == '/albums'
               && core_url.substr(core_url.length - 5) == '.json') {
        handle_get_album(req, res);
    } else {
        send_failure(res, 404, invalid_resource());
    }
}

The serve_page function actually determines what page the client has asked for, fixes up the basic.html template file, and sends the resulting HTML skeleton down to the browser:

/**
 * All pages come from the same one skeleton HTML file that
 * just changes the name of the JavaScript loader that needs to be
 * downloaded.
 */
function serve_page(req, res) {
    var page = get_page_name(req);

    fs.readFile('basic.html', (err, contents) => {
        if (err) {
            send_failure(res, 500, err);
            return;
        }

        contents = contents.toString('utf8');

        // replace page name, and then dump to output.
        contents = contents.replace('{{PAGE_NAME}}', page);
        res.writeHead(200, { "Content-Type": "text/html" });
        res.end(contents);
    });
}

function get_page_name(req) {
    var core_url = req.parsed_url.pathname;
    var parts = core_url.split("/");
    return parts[2];
}

Instead of using fs.createReadStream, you use fs.readFile, which reads in the entire contents of a file to a Buffer. You have to convert this Buffer to a string, and then you can fix up the contents so the right JavaScript bootstrapper is indicated. This approach is not recommended for large files because it wastes a lot of memory, but for something as short as this basic.html file, it’s perfectly fine and convenient!

The JavaScript Loader/Bootstrapper

The HTML pages are going to be quite simple, so you can start with a reasonably trivial JavaScript bootstrapper. As you add more functionality later in the book, you make it a bit more complicated, but Listing 6.2 shows the one you can start with now, which you save as home.js.

Listing 6.2 The JavaScript Page Loader (home.js)


$(function(){

    var tmpl,   // Main template HTML
    tdata = {};  // JSON data object that feeds the template

    // Initialize page
    var initPage = function() {

        // Load the HTML template
        $.get("/templates/home.html", function(d){                     // 1
            tmpl = d;
        });

        // Retrieve the server data and then initialize the page
        $.getJSON("/albums.json", function (d) {                       // 2
            $.extend(tdata, d.data);
        });

        // When AJAX calls are complete parse the template
        // replacing mustache tags with vars
        $(document).ajaxStop(function () {
            var renderedPage = Mustache.to_html(tmpl, tdata);          // 3
            $("body").html(renderedPage);
        })
    }();
});


The $(function () { ... syntax is basically the same as $(document).ready(function () ... in jQuery, just a bit shorter. So, this function is called after all the other resources (most notably jQuery and Mustache) are loaded. It then does the following three things:

1. It requests the template file, home.html, from the server by calling the URL /templates /home.html.

2. It asks for the JSON data representing albums in the application, /albums.json.

3. Finally, it gives these two things to Mustache to do its templating magic.

Templating with Mustache

The template engine I chose for the photo album app, Mustache, is easy to use, fast, and small. Most interestingly, it has only tags, wrapped in {{ and }}, known as mustaches. It does not have any if statements or looping constructs. Tags act as a set of rules that play out depending on the data you provide.

The basic usage of Mustache, which you saw in the preceding sections, is

var html_text = Mustache.to_html(template, data);
$("body").html(html_text);

To print properties from an object, you use the following:

Mustache template:

The album "{{name}}" has {{photos.length}} photos.

JSON:

{
  "name": "Italy2012",
  "photos" : [ "italy01.jpg", "italy02.jpg" ]
}

Output:

The album "Italy2012" has 2 photos.

To apply a template over every item in a collection or array, you use the # character. For example:

Mustache:

{#albums}}
   * {{name}}
{{/albums}}

JSON:

{
  "albums" : [ { "name" : "italy2012" },
               { "name" : "australia2010" },
               { "name" : "japan2010" }]
}

Output:

* italy2012
* australia2010
* japan2010

If you do not have any matching results, nothing is printed. You can capture this case by adding the ^ character:

Mustache:

{{#albums}}
   * {{name}}
{{/albums}}
{{^albums}}
   Sorry, there are no albums yet.
{{/albums}}

JSON:

{ "albums" : [ ]  }

Output:

Sorry, there are no albums yet.

If the object after the # character isn’t a collection or array but just an object, the values from it are used without iterating:

Mustache:

{{#album}}
  The album "{{name}}" has {{photos.length}} photos.
{{/album}}

JSON:

{
  "album": {  "name": "Italy2012",
              "photos" : [ "italy01.jpg", "italy02.jpg" ] }
}

Output:

The album "Italy2012" has 2 photos.

By default, all values are HTML escaped; that is, < and > are replaced with &lt; and &gt; respectively, and so on. To tell Mustache not to do this, use the extra hairy Mustache triple brackets, {{{ and }}}:

Mustache:

{{#users}}
   * {{name}} says {{{ saying }}}
     * raw saying: {{ saying }}
{{/users}}

JSON:

{
  "users" : [ { "name" : "Marc", "saying" : "I like <em>cats</em>!" },
              { "name" : "Bob", "saying" : "I <b>hate</b> cats" },
}

Output:

  * Marc says I like <em>cats</em>!
    * raw saying: I like &lt;em&gt;cats&lt;/em&gt;!
  * Bob says I <b>hate</b> cats
    * raw saying: I  &lt;b&gt;hate&lt;/b&gt; cats

Your Home Page Mustache Template

Now, it’s time to write your first template for your home page, which you can save in home.html, shown in Listing 6.3.

Listing 6.3 The Home Page Template File (home.html)


<div id="album_list">
  <p> There are {{ albums.length }} albums</p>
  <ul id="albums">
    {{#albums}}
      <li class="album">
        <a href='http://localhost:8080/pages/album/{{name}}'>{{name}}</a>
      </li>
    {{/albums}}
    {{^albums}}
      <li> Sorry, there are currently no albums </li>
    {{/albums}}
  </ul>
</div>


If you have any albums, this code iterates over each of them and provides an <li> element with the name of the album and a link to the album page, accessed via the URL /page/album/album_name. When there are no albums, a simple message is printed instead.

Putting It All Together

I threw a lot of new stuff at you in the preceding sections, so stop now and make this all run. You should now have the following file layout for your web application:

+ project_root/
    package.json                              // We need async 2.x
    server.js                                 // GitHub source
    basic.html                                // Listing 6.1
    + content/
        home.js                               // Listing 6.2
        album.js                              // Listing 6.5
        jquery-1.8.3.min.js
        mustache.js
        style.css
    + templates/
        home.html                             // Listing 6.3
        album.html                            // Listing 6.4
+ albums/
        + whatever albums you want

The code for home.html and album.html, which we have not previously seen, can be found in Listings 6.4 and 6.5.

To run this project, you should be able to type

node server.js

You can then switch to a web browser and browse to http://localhost:8080/page/home. If you want to see what happens on the command line, try using curl as follows:

curl -i -X GET http://localhost:8080/page/home

Because curl executes no client-side JavaScript for you, the template and JSON date are not loaded via Ajax, and you get only the basic HTML skeleton from basic.html.

Listing 6.4 Another Mustache Template Page (album.html)


<div id="album_page">
  {{#photos}}
  <p> There are {{ photos.length }} photos in this album</p>
  <div id="photos">
      <div class="photo">
        <div class='photo_holder'>
          <img style='width: 250px; height: 250px; background-color: #fbb'
               src="{{url}}" border="0"/></div>
        <div class='photo_desc'><p>{{ desc }}</p></div>
      </div>
  </div> <!-- #photos -->
  <div style="clear: left"></div>
  {{/photos}}
  {{^photos}}
      <p> This album doesn't have any photos in it, sorry.<p>
  {{/photos}}
</div> <!-- #album_page -->


Listing 6.5 Our Album Page Bootstrapper JavaScript File (album.js)


$(function(){

    var tmpl,   // Main template HTML
    tdata = {};  // JSON data object that feeds the template

    // Initialize page
    var initPage = function() {

        // get our album name.
        parts = window.location.href.split("/");
        var album_name = parts[5];

        // Load the HTML template
        $.get("/templates/album.html", function(d){
            tmpl = d;
        });

        // Retrieve the server data and then initialize the page
        $.getJSON("/albums/" + album_name + ".json", function (d) {
            var photo_d = massage_album(d);
            $.extend(tdata, photo_d);
        });

        // When AJAX calls are complete parse the template
        // replacing mustache tags with vars
        $(document).ajaxStop(function () {
            var renderedPage = Mustache.to_html( tmpl, tdata );
            $("body").html( renderedPage );
        })
    }();
});

function massage_album(d) {
    if (d.error != null) return d;
    var obj = { photos: [] };

    var af = d.data.album_data;

    for (var i = 0; i < af.photos.length; i++) {
        var url = "/albums/" + af.short_name + "/" + af.photos[i].filename;
        obj.photos.push({ url: url, desc: af.photos[i].filename });
    }
    return obj;
}


If you’ve done everything correctly, you should see output something similar to that shown in Figure 6.2. It’s not pretty, but now you can start to play around with the template file and the style.css file to really make the application yours!

Image

Figure 6.2 The template-enabled JSON application running in the browser

Summary

This chapter introduced two key Node.js features, streams and events, that you used to serve up static content from your web application. Equally importantly, this chapter introduced client-side template engines, and you used Mustache to put together a simple start to a front end for your photo album application.

You might have noticed, however, that some of the things you have been doing thus far seem unnecessarily complicated, tedious, or possibly even worryingly bug-prone (the handle_incoming_request function in particular). It would be nice if there were some further modules to help you keep track of things more easily.

It is thus that you are not going to develop the photo album much further in its current form but continue to look at some modules that will clean up your code significantly and allow you to add features quickly and with little code. You will do so using the Express Application Framework for Node.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset