7. Building Web Applications with Express
Thus far, you have been learning the fundamentals and core concepts of Node.js; armed with these ideas, you have built some simple applications, although you’ve had to write a lot of code to do some reasonably basic things. It’s time to change gears a bit and start building more interesting web applications, using one of the biggest strengths of Node: the huge collection of libraries and modules available to you through npm. Now that you understand the core workings of Node and modules, it’s time to start finding ways to make your life significantly easier and your development much faster.
In this chapter, you start working with express, a web application framework for Node. It takes care of many of the basic tasks you learned about in previous chapters and even exposes some powerful new functionality that you can take advantage of in your apps. I show you how to build apps with express and update your photo album app to use it. Then you look a bit more at application design, learn how to expose the API from your JSON server, and explore some new functionality you haven’t seen before.
Installing Express
In this chapter, you are going to slowly change the way you install applications via npm. You might expect that, to install express, you’d just type
npm install express
and this would work just fine. However, it always installs the latest version of the express node module, which you might not want when you’re deploying your application to live production servers. When you develop and test against a particular version, it’s good to have that same version run on the live servers. Only after you update to a newer version and fully test against that version do you want to use it on the live servers.
So, the basic methodology for building Node applications is now as follows:
1. Create an application folder.
2. Write a package.json file describing the package and stating which npm modules are needed.
3. Run npm install to make sure these modules are installed properly.
So, create a new folder called trivial_express/ for your express test application and then put the following into package.json in that folder:
{
"name": "Express-Demo",
"description": "Demonstrates Using Express and Connect",
"version": "0.0.1",
"private": true,
"dependencies": {
"express": "4.x"
}
}
The name and description fields are reasonably self-explanatory, and the app is just a toy right now, so it has an appropriate version number. As you saw in Chapter 5, “Modules,” you can set private to true, which tells npm never to try installing it in the live npm registry because it is just a test project for your own use.
After creating the file, you can run the following to read in the dependencies section and install all the appropriate modules:
npm install
Because express itself depends on many different modules, you see a decent amount of output before it exits, but then your node_modules/ folder should be ready to go. If you ever make a change to your package.json, you can just run npm update to make sure all the modules are correctly installed.
Hello World in Express
Recall from Chapter 1, “Getting Started,” that your first web server in Node.js was roughly like the following:
var http = require("http");
function process_request(req, res) {
res.end("Hello World");
}
var s = http.createServer(process_request);
s.listen(8080);
Well, in express, it looks quite similar:
var express = require('express');
var app = express();
app.get('/', function(req, res){
res.end('hello world');
});
app.listen(8080);
Save that, run it, and then either run curl localhost:8080 or enter localhost:8080 in the web browser, and you should get the hello world output you expect. One of the great things about express is that it is built on top of everything you have seen so far, so the HTTP request and response objects have the same methods and properties they had before. To be certain, they have many new things added that make them more interesting and powerful, but you are not operating in a completely new world here—just an extended version of the one with which you are already familiar and comfortable!
Routing and Layers in Express
Express implements a system known as middleware that provides a collection of common functions that network applications might need. While previous versions of Express used a Node module called connect to implement this, versions starting with 4.0 no longer do so—it is now built in to Express.
At the heart of this middleware model is the concept of a flow of functions, or what some people like to describe as a layered set of functions.
You set up your application by creating a sequence of functions (layers) through which express navigates. When one decides that it wants to take over, it can complete the processing and stop the sequence. Express then walks back up through the sequence to the end (some layers do their process “on the way in”; others “on the way out”).
So, in the preceding express application, there is only one layer in the sequence:
app.get('/', function(req, res){ });
This function, which is a helper provided by express, processes the incoming request (by calling the provided function) if the following are both true:
The HTTP request method is GET.
The requested URL is /.
If the incoming request does not match these criteria, the function is not called. If no function matches the requested URL—for example, browse or curl to /some/other/url—express just gives a default response:
Cannot GET /some/other/url
Routing Basics
The general format of the URL routing functions is as follows:
app.method(url_pattern, optional_functions, request_handler_function);
The preceding route handler is for handling GET requests for the URL /. To support handling a POST request on a particular URL, you can use the post method on the express application object:
app.post("/forms/update_user_info.json", function (req, res) { ... });
Similarly, express supports DELETE and PUT methods, both of which you use in the photo-sharing application, via the delete and put methods, respectively. Alternately, you can use the all method to indicate that the routing function should accept the given URL with any given HTTP method.
app.all("/users/marcwan.json", function (req, res) { ... });
The first parameter to the routing function is a regular expression to match an incoming request URL. It can be simple as follows:
/path/to/resources
Or it can contain regular expression features, such as
/user(s)?/list
which matches both /users/list and /user/list, while
/users/*
matches anything starting with /users/.
One of the most powerful features of routing is the ability to use placeholders to extract named values from the requested route, marked by the colon (:) character. When the route is parsed, express puts the matched placeholders on the req.params object for you. For example:
app.get("/albums/:album_name.json", function (req, res) {
res.end("Requested " + req.params.album_name);
});
If you call the preceding app requesting /albums/italy2012.json, you get the following output:
Requested italy2012.
You can put multiple parameters in the same URL as well:
app.get("/albums/:album_name/photos/:photo_id.json", function (req, res) { ... );
If your function is called, req.params has both album_name and photo_id set with the appropriate incoming values. Placeholders match any sequence of characters except for forward slashes.
The function you provide to the routing method is given the request and response object, as you have seen. However, it is also given a third parameter, which you are free to use or ignore. It is usually labeled next (the flow of the layered functions is always continued through functions called next), and gives you the opportunity to do some additional examinations on the incoming URL and still choose to ignore it, as follows:
app.get("/users/:userid.json", function (req, res, next) {
var uid = parseInt(req.params.userid);
if (uid < 2000000000) {
next(); // don't want it. Let somebody else process it.
} else {
res.end(get_user_info(uid));
}
});
Updating Your Photo Album App for Routing
The photo album app is quite easy to adapt to express. You have to do only a few things to get it working:
Create a package.json and install express.
Replace the http module’s server with that of express.
Replace the handle_incoming_request function with routing handlers.
Update the way you get query parameters in express (they are placed on req.query for your convenience).
Now copy the last version of the photo app you created at the end of Chapter 6, “Expanding Your Web Server,” to a new folder (you can call it basic_routing/) and place the following package.json inside:
{
"name": "Photo-Sharing",
"description": "Our Photo Sharing Application",
"version": "0.0.2",
"private": true,
"dependencies": {
"async": "2.x",
"express": "4.x"
}
}
You can run npm install to make sure express and async are installed to node_modules/. You can then change the top of the server.js file to use express instead of the http module:
var express = require('express');
var app = express();
var path = require("path"),
async = require('async');
fs = require('fs');
And you change
http.listen(8080);
to
app.listen(8080);
Next, you have to replace the handle_incoming_request function, which looks as follows:
function handle_incoming_request(req, res) {
req.parsed_url = url.parse(req.url, true);
var core_url = req.parsed_url.pathname;
// test this fixed url to see what they're asking for
if (core_url.substring(0, 7) == '/pages/') {
serve_page(req, res);
} else if (core_url.substring(0, 11) == '/templates/') {
serve_static_file("templates/" + core_url.substring(11), res);
} else if (core_url.substring(0, 9) == '/content/') {
serve_static_file("content/" + core_url.substring(9), res);
} else if (core_url == '/albums.json') {
handle_list_albums(req, res);
} else if (core_url.substr(0, 7) == '/albums'
&& core_url.substr(core_url.length - 5) == '.json') {
handle_get_album(req, res);
} else {
send_failure(res, 404, invalid_resource());
}
}
You replace it with these routing functions:
app.get('/albums.json', handle_list_albums);
app.get('/albums/:album_name.json', handle_get_album);
app.get('/content/:filename', function (req, res) {
serve_static_file('content/' + req.params.filename, res);
});
app.get('/templates/:template_name', function (req, res) {
serve_static_file("templates/" + req.params.template_name, res);
});
app.get('/pages/:page_name', serve_page);
app.get('*', four_oh_four);
function four_oh_four(req, res) {
send_failure(res, 404, invalid_resource());
}
Not only do the routing functions seem much cleaner and simpler than the previous function you had, but it is much easier to see the URLs that will be matched and how the app will function. You added a new route matching "*" so that all other requests will be given a 404 response.
There is one problem with the routing functions right now, however. What happens if the user asks for /pages/album/italy2012? Currently, the app will fail: the regular expression matching for parameters never includes forward slash (/) characters, so the route /pages/:page_name does not match. To solve this problem, you must add the extra route:
app.get('/pages/:page_name/:sub_page', serve_page);
Now all the pages will route to the correct function, and you have also set yourself up to quickly identify the subpage being requested.
One of the nice things about the app now is that you no longer need to parse the incoming requests or parameters; express does all this for you. So, whereas before you had to include the url module to parse the incoming URL, it’s now ready for you right when you get to the routing functions. Similarly, you never need to parse GET query parameters because they are also prepared for you in advance. So you can update the small helper functions as follows:
function get_album_name(req) {
return req.params.album_name;
}
function get_template_name(req) {
return req.params.template_name;
}
function get_query_params(req) {
return req.query;
}
function get_page_name(req) {
return req.params.page_name;
}
With those changes, the photo-sharing app should be updated and ready to go with express. It should behave exactly like the old one.
REST API Design and Modules
When you are designing JSON servers such as the one you are writing, it is important to spend some time thinking about the API and how clients will use it. Spending some time up front to design a good API helps you to think about how people will use the application and also helps you organize and refactor the code to reflect this conceptual understanding.
API Design
For the photo-sharing JSON server, you are developing what’s called a RESTful JSON server. The word REST comes from Representational State Transfer, and basically implies that you can request an accurate representation of an object from the server. REST APIs focus on four core operations (which, coincidentally, map to four HTTP request methods):
Creation (PUT)
Retrieval (GET)
Updating (POST)
Destruction (DELETE)
Some people refer to these operations as CRUD, and pretty much everything you do in your API centers around doing one of these things to objects, whether they are albums, photos, comments, or users. Although it is not possible to design the perfect API, I will do my absolute best to help you design one that is at least extremely intuitive and doesn’t leave the client authors scratching their heads as to what you intended. Here are some principles to follow when designing RESTful interfaces:
There are two basic kinds of URLs. Most URLs you design will be a variation on these two:
1. Collections—for example, /albums.
2. Specific items within these collections—for example, /albums/italy2012.
Collections should be nouns, specifically plural nouns, such as albums, users, or photos.
PUT /albums.json—The HTTP request body contains the JSON with the data for the new album.
You fetch or update an object by specifying the specific instance on the collection with GET or POST, respectively:
GET /albums/italy2012.json—The request returns this album.
POST /albums/italy2012.json—The HTTP request body contains the JSON with the new data for this album.
You destroy objects with DELETE.
DELETE /albums/italy2012.json—Destroys this album.
If you have collections off collections, for example, photos associated with albums, you just continue the pattern:
GET /albums/italy2012/photos.json—Returns all the photos in this album.
PUT /albums/italy2012/photos.json—Adds a new photo to this collection.
GET /albums/italy2012/photos/23482938424.json—Fetches the photo with the given ID.
Slight changes to the way in which you get data from these collections, such as pagination or filtering, should all be done via GET query parameters:
GET /albums.json?located_in=Europe—Fetches only those albums in Europe.
GET /albums/japan2010/photos.json?page=1&page_size=25—Fetches the given page of 25 photos for the japan2010 album.
You assign a version number to the API so that if you want to make major changes to it in future versions that would not be compatible, you can simply update the version number. You prefix the API URLs with /v1/ for this first version.
Finally, you suffix all your URLs that return JSON with .json so that clients know what data will be in the responses. In the future, you could also add support for .xml or whatever other formats you want.
With this in mind, the new API for the photo-sharing application centers around albums. It looks as follows:
/v1/albums.json
/v1/albums/:album_name.json
/pages/:page_name
/templates/:template_name
/content/:filename
Why don’t you put version numbers on the static content URLs? Because they’re not part of the JSON server interface, but more helpers to provide the contents for the web browser client. Users always get the latest version of this and thus get code that knows how to use the correct version of the API.
It should be trivial for you to update the server.js file with this API design change. Don’t forget to also update the bootstrapping JavaScript files in /content/ with the new URLs as well! Give that a try before continuing onward.
Modules
With the APIs updated around the new REST interface, you can take advantage of some of the clarity you have given to the application. You now have clear functionality boundaries around albums and pages, so you can put those areas of functionality into their own modules.
Create a subfolder of the application folder called handlers/ and put the new modules in there. For albums, create a file in that folder called albums.js and move the four album manipulation functions into that file. You can view the source for this project on GitHub under the folder named handlers_as_modules/. Effectively, all you are doing is taking the code for the functions handle_get_album and handle_list_albums and putting them, along with their accompanying helper functions load_album and load_album_list, into the new module. Take a look at this for a minute and see how easy it is to reorganize the code quickly.
Recall that the server.js file has a few functions to help you extract album names and query parameters and the like. You can get rid of them and just put the appropriate member references for req.params and req.query in the right place.
There are also a few functions to help you simplify sending success and error codes, such as send_success, send_failure, and invalid_resource. You can put them into their own module called helpers.js in the handlers/ directory. You refer to this at the top of the albums.js file, and the contents are shown in Listing 7.1.
Listing 7.1 Helper Functions (helpers.js)
exports.version = '0.1.0';
exports.make_error = function(err, msg) {
var e = new Error(msg);
e.code = err;
return e;
}
exports.send_success = function(res, data) {
res.writeHead(200, {"Content-Type": "application/json"});
var output = { error: null, data: data };
res.end(JSON.stringify(output) + "
");
}
exports.send_failure = function(res, server_code, err) {
var code = (err.code) ? err.code : err.name;
res.writeHead(server_code, { "Content-Type" : "application/json" });
res.end(JSON.stringify({ error: code, message: err.message }) + "
");
}
exports.invalid_resource = function() {
return make_error("invalid_resource",
"the requested resource does not exist.");
}
exports.no_such_album = function() {
return make_error("no_such_album",
"The specified album does not exist");
}
Now, when you want to refer to the album functions, you add the following require to the top of server.js:
var album_hdlr = require('./handlers/albums.js');
And then you can update the routing handlers for the album functions to be as follows:
app.get('/v1/albums.json', album_hdlr.list_all);
app.get('/v1/albums/:album_name.json', album_hdlr.album_by_name);
Similarly, you can move the functionality to build pages into the handlers/ directory in a file called pages.js, as in Listing 7.2. This task really only involves moving the serve_page function.
Listing 7.2 Building Pages (pages.js)
var helpers = require('./helpers.js'),
fs = require('fs');
exports.version = "0.1.0";
exports.generate = function (req, res) {
var page = req.params.page_name;
fs.readFile(
'basic.html',
function (err, contents) {
if (err) {
send_failure(res, 500, err);
return;
}
contents = contents.toString('utf8');
// replace page name, and then dump to output.
contents = contents.replace('{{PAGE_NAME}}', page);
res.writeHead(200, { "Content-Type": "text/html" });
res.end(contents);
}
);
};
You can then update the routing handler for pages, as follows:
var page_hdlr = require('./handlers/pages.js');
app.get('/v1/pages/:page_name', page_hdlr.generate);
After all is said and done, the server.js file has left only the functions for serving static content, which you can view in the GitHub source for Chapter 7 in the handlers_as_modules/ project folder.
The application should have the same functionality as before but is now much more modularized and is easier to read and follow along. You now also have an idea of how you can add functionality to it as you proceed.
Additional Middleware Functionality
I’ve mentioned before that express implements what is called a middleware library. These components chain together and are given every incoming request one at a time until one of them decides to process it and ceases calling the provided next function. The routing handlers form part of this chain. Express used to come with quite a large number of other useful components built in, but most have been split out into separate npm modules now. Let’s spend a while reviewing some of the more interesting ones that we’ll make use of.
Usage
You use middleware components in express with the use method. For example, there is a great middleware package called morgan that will log all requests to your app in various configurable ways. To use it, you would write the following:
var express = require('express'),
morgan = require('morgan');
var app = express();
app.use(morgan('dev'));
/* etc. ... */
To use this, we’d need to add
"morgan": "1.x",
to the dependencies section of our package.json file. The morgan function that you call returns a function that will form part of the filter layering in your application.
Express comes with a few of components built in, but you are allowed to use any compatible middleware component. You can use any of those in the following manner, provided you’ve added it to the dependencies section of your package.json file:
var express = require('express');
var compression = require('compression');
var app = express();
app.use(compression());
/* etc. ... */
Or you can download third-party components from npm or somewhere else and install them directly:
var express = require('express');
var middle_something = require('middle_something');
var app = express();
app.use(middle_something());
/* etc. ... */
Configurations
Using some middleware only in certain configurations can be helpful. Some might modify your output for speed or size so that it can be difficult or annoying to use curl to test your server. To do this, you call the configure method on the express app, providing it with the name of the configuration you want to target and a function that will be called when that configuration is needed. If you don’t provide a name, that function is called always, and you can provide multiple configuration names separated by a comma before you provide the function, as shown here:
app.configure(function () {
app.use(bodyParser());
});
app.configure('dev', function () {
app.use(morgan('dev'));
});
app.configure('production', 'staging', function () {
app.use(morgan());
});
Middleware or routes that are set up outside configure function calls are considered applicable for all configurations.
To run your app with a particular configuration, set the NODE_ENV environment variable to the one you want. On most shells on your Mac or UNIX computer, you can run node as follows:
NODE_ENV=production node program.js
On Windows, you can, in the command prompt, just set the NODE_ENV and then run node:
set NODE_ENV=production
node program.js
Ordering of Middleware
The layering and order of these middleware components is important. As I mentioned, express and connect work through them one at a time until it finds one that decides it can handle the request. To see how this is relevant, consider the following simple app:
var express = require('express'),
morgan = require('morgan');
var app = express();
app.use(express.logger('dev'))
// move this to AFTER the next use() and see what happens!
.use(express.responseTime())
.use(function(req, res){
res.end('hello world
');
})
.listen(8080);
Call this app by using
curl -i localhost:8080/blargh
You should see the following:
HTTP/1.1 200 OK
X-Powered-By: Express
X-Response-Time: 0ms
Date: Wed, 05 Dec 2012 04:11:43 GMT
Connection: keep-alive
Transfer-Encoding: chunked
hello world
Now, watch what happens if you move the responseTime middleware component to the end, as follows:
var express = require('express');
var app = express();
app.use(express.logger('dev'))
// move this to AFTER the next use() and see what happens!
.use(function(req, res){
res.end('hello world
');
})
.use(express.responseTime())
.listen(8080);
Notice that the X-Response-Time header is gone!
HTTP/1.1 200 OK
X-Powered-By: Express
Date: Wed, 05 Dec 2012 04:14:05 GMT
Connection: keep-alive
Transfer-Encoding: chunked
hello world
The function that prints “Hello World!” took the request and handled it fully by not calling next and by using res.end to close down the response, so the responseTime middleware never got a chance to inspect the request. As you’re adding middleware components, think about how they’re going to fit together for the various routes your app will be using.
Static File Handling
You have written a little bit of Node code to serve static files in your application, but it turns out that doing so is largely unnecessary. Express provides a static middleware component that can do all this work for you!
To use it, you just pass the name of the base folder whose contents you want to serve statically to the middleware creation function, as follows:
app.use(express.static("/secure/static_site_files"));
You put this before the URL routers, and when a request comes in, this static layer takes the URL, appends it to its base folder name, and looks to see whether that file exists. For example, if you request /content/style.css, this middleware looks to see whether /secure/static_site_files/content/style.css exists and is readable. If it is, the middleware serves that file and stops calling further layers. If not, it calls the next function.
If you want to serve static content from multiple locations, add multiple components:
app.use(express.static("/secure/core_site_content"));
app.use(express.static("/secure/templates_and_html"));
Note that while you technically can serve content from your application’s folder by using the __dirname predefined variable in Node (it gives you the path to the currently executing Node script), this is a terrible idea and you should never do it.
The static middleware provides no security at all, so say you take the photo-sharing application from before and just add
app.use(express.static(__dirname));
You could then—as expected—browse for /content/style.css and /templates/home.html without any problems. The problem is that if the user asks for /server.js or /handlers/albums.js, the static middleware happily serves that as well because they both lie off the __dirname base folder.
You might ask why security has not been added to the static middleware component, but this is the wrong question—it’s an unnecessary complication. If you want to use this component, put your static content outside your application source tree. Indeed, as I’ve hinted before, you might even choose to have your application avoid serving static content altogether and opt to put the static content on CDNs instead. So get used to separating those components out from your source tree right from the start.
If you rewrite the server.js file to use the static middleware, it looks something like the following. It does, require, however, that you move the content/, templates/, and album/ folders to another location so that you can avoid the security problem I just mentioned. The updated app structure looks as follows:
+ root_folder/
+ static/
+ albums/
+ content/
+ templates/
+ app/
+ handlers/
+ node_modules/
Then the server.js file can look like that shown in Listing 7.3. One nice thing about this new version of the server is that you’ve managed to remove two routing functions—those for content and templates.
Listing 7.3 Using the Static Middleware (server.js)
var express = require('express');
var app = express();
var fs = require('fs'),
album_hdlr = require('./handlers/albums.js'),
page_hdlr = require('./handlers/pages.js'),
helpers = require('./handlers/helpers.js');
app.use(express.static(__dirname + "/../static"));
app.get('/v1/albums.json', album_hdlr.list_all);
app.get('/v1/albums/:album_name.json', album_hdlr.album_by_name);
app.get('/pages/:page_name', page_hdlr.generate);
app.get('/pages/:page_name/:sub_page', page_hdlr.generate);
app.get("/", function (req, res) {
res.redirect("/pages/home");
res.end();
});
app.get('*', four_oh_four);
function four_oh_four(req, res) {
res.writeHead(404, { "Content-Type" : "application/json" });
res.end(JSON.stringify(helpers.invalid_resource()) + "
");
}
app.listen(8080);
If you’re paying attention, you might have noticed the new route added for / that reroutes you to /pages/home using the ServerResponse.redirect method to save some typing in the web browser!
POST Data, Cookies, and Sessions
Express already provides you with some help with the query parameters by parsing them and placing them in req.query. Although you saw back in Chapter 4, “Writing Applications,” that it’s possible to load and parse form POST data yourself using streams, the good news is that you don’t have to do this—middleware can do all this work for you. And you can even have it set up cookies and sessions for you. The middleware component modules to do this are body-parser (1.x), cookie-parser (1.x), and express-session (1.x), respectively:
var bodyParser = require('body-parser'),
cookieParser = require('cookie-parser');
// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }))
// parse application/json
app.use(bodyParser.json())
app.use(cookieParser());
For the bodyParser, you specify which formats you want to support, and then they are parsed and placed in req.body as an object. cookieParser does something similar, except it places the values in req.cookies instead and needs no configuration at all.
To set a cookie on the outgoing response, use the cookie function on the response object, with the name and value (and optional timeout values):
var express = require('express'),
morgan = require('morgan');
cookieParser = require('cookie-parser');
var app = express()
.use(morgan('dev'))
.use(cookieParser())
.use(function(req, res){
res.cookie("pet", "Zimbu the Monkey",
{ expires: new Date(Date.now() + 86400000) });
res.end(JSON.stringify(req.query) + "
");
})
.listen(8080);
To clear a cookie you’ve set, just use the clearCookie method on the response object with the name of the cookie you’d like removed.
Setting up sessions is a bit more involved, requiring that you first provide a few variables to configure everything:
.use(session({ secret: "blargleipoajsdfoiajf",
resave: false,
saveUninitialized: true,
cookie: { maxAge: 1800000 } }))
In this example, you set a secret key, and then tell the session middleware it’s not necessary to force a resave if nothing has changed during a request. We also tell it to save the session data at the end of a request even if nothing has been set. Finally, we set an age for the session cookie (about three weeks here). If you want to expire them, provide the number of milliseconds of validity. Here, the code indicates you will not accept session cookies older than 30 minutes.
When you have the sessions ready to go, req.session is populated with the session data. To add new session data, you simply set a new value on this object. It is persisted and sent with the response:
var express = require('express'),
morgan = require('morgan'),
cookieParser = require('cookie-parser'),
session = require('express-session');
var app = express()
.use(morgan('dev'))
.use(cookieParser())
.use(session({ secret: "blargleipoajsdfoiajf",
resave: false,
saveUninitialized: true,
cookie: { maxAge: 1800000 } }))
.use(function (req, res){
var x = req.session.last_access;
req.session.last_access = new Date();
res.end("You last asked for this page at: " + x);
})
.listen(8080);
File Uploads
The body-parser middleware component lets you parse data sent along with POST requests. Combined with the multer middleware component (add version 1.x to your package.json), you can additionally accept multipart bodies that include uploaded files and similar things. If you upload a file to your app using the multipart/form-data enctype, you see req.file or req.files contain any files sent with the request, depending on whether you told multer to expect a single file or multiple files (it can handle both!):
var express = require('express'),
morgan = require('morgan'),
multer = require('multer'); // includes body-parser
var upload = multer({ dest: "ul/" }); // 1
var app = express()
.use(morgan('dev'));
app.post('/uptest', upload.single("album_cover"), function (req, res) { // 2
console.log("BODY: " + JSON.stringify(req.body, 0, 2));
console.log("FILE: " + JSON.stringify(req.file, 0, 2));
if (!req.file || req.file.fieldname != 'album_cover') {
res.end("Hunh. Did you send a file?
");
} else {
res.end("You have asked to set the album cover for "
+ req.body.albumid
+ " to '" + req.file.originalname + "'
");
}
});
app.listen(8080);
To test this, you can still use curl! It can upload files along with the request by combining the --form (-F) option (for form data) and the @ operator with a filename, which tells it to send that file:
curl -i -H "Expect:" --form '[email protected]'
--form albumid=italy2012 http://localhost:8080/uptest
There is one trick, however, and that is that curl sometimes expects the server to send the response 100 Continue and then hang waiting. To get around this, you pass -H "Expect:" to curl, which lets you test file uploads normally.
In the example above, after we’ve initialized the multer module, we have to configure it, which you see in // 1. We create an object called upload, and indicate that we want all uploaded files to go in a folder called ul/. Of course, you should choose a path that is isolate and doesn’t leave you open to a DoS attack (imagine if somebody just kept uploading files until your hard disk was full!).
Next we tell multer that we want to allow a single file upload with the field name album_cover with the POST request /uptest. This will be placed in req.file. We could instead tell multer we wanted to accept multiple files by using upload.array("field_name") (expect an array of files all with the field field_name), or upload.fields("field1", "field2", etc.) (expect multiple files with the specified field names). For these, we would have req.files with a list of all files that were parsed from the incoming request.
Whenever you’re using multer or just the body-parser middleware modules on its own, you’ll have req.body parsed and ready for you to use in your code.
Better Browser Support for PUT and DELETE
PUT (object creation) and DELETE (object destruction) are critical parts of the REST APIs that you are using, and the client web pages send data to the server via these methods. For example, to say you want to delete a comment in jQuery, you use
$.ajax({
url: 'http://server/v1/comments/234932598235.json',
type: 'DELETE'
}
In most modern web browsers, this technique works just fine. However, in some older browsers (and in any Internet Explorer version before 10), it fails. The reason is that their implementation of XmlHttpRequest, the object via which AJAX requests are done, does not support PUT or DELETE as a method.
A common and elegant solution is to add a new header called X-HTTP-Method-Override to your output, with the method you really want to use, and then just use POST to post the request to the server. The jQuery code would then become
$.ajax({
beforeSend: function(xhr) {
xhr.setRequestHeader('X-HTTP-Method-Override', 'DELETE');
},
url: 'http://server/v1/comments/234932598235.json',
type: 'POST'
}
To support this solution on your server, you use the method-override middleware, as follows:
var methodOverride = require('method-override'); // add v2.x to package.json!
app.use(methodOverride());
It looks for this header and converts POST requests to DELETE or PUT as necessary. Thus, the app.delete and app.put routes work exactly as expected.
Compressing Output
To save on bandwidth costs and speed, many web servers compress output using gzip or some similar algorithm before sending it down to the client. This requires that the client first indicate that it is able to accept compressed output with the Accept-Encoding HTTP header. If this is provided and the server knows how, it adds the Content-Encoding header on output with the appropriate algorithm specified and then sends down the compressed data.
To use this feature in your app, you can use the compression middleware. It is truly a hassle-free component; it checks whether the client accepts compressed output via headers, sets the response headers, and compresses the output, as shown here:
var express = require('express'),
compression = require('compression');
var app = express();
app.use(express.logger('dev'));
app.use(compression());
app.get('/', function(req, res){
res.end('hello world this should be compressed
');
});
app.listen(8080);
The only downside to this module is that it might interfere with some of your development testing (curl just prints the compressed binary data to the screen, which can be annoying), so it is common for you to use this only in a production configuration, as follows:
var express = require('express'),
morgan = require('morgan'),
compression = require('compression');
var app = express();
app.use(morgan('dev'));
app.configure('production', function () {
app.use(compression());
});
app.get('/', function(req, res){
res.end('hello world this may or may not be compressed
');
});
app.listen(8080);
Adding Authentication to our Application
Pretty much every interesting application being written today requires authentication of some sort. Doing this from scratch on your own can be a hassle—not only do you have to write a lot of code, but it’s also error prone and can lead to serious security holes.
Fortunately, there exists a module for Node.js that can handle a vast majority of this for us, called passportjs. or just passport.
In this section, we’re going to create a simple application that supports authentication and shows off all the key parts of passport that we’ll want to add to our applications.
Passport supports a huge variety of authentication schemes, ranging from storing usernames and passwords in your local application to authentication schemes used by Facebook, Twitter, and Google, as well as supporting generic OAuth 2.0 authentication.
Getting Started
Let’s build an application that authenticates requests with passport by using local storage for usernames and passwords (we might put these in a database, for example). For the purposes of this sample, we’ll just hardcode the acceptable combinations directly in our app.
To start, we’ll need to include a number of modules in our package json as follows:
{
"name": "authtest",
"description": "Testing passport",
"version": "0.0.1",
"private": true,
"dependencies": {
"body-parser": "1.x",
"cookie-parser": "1.x",
"express": "4.x",
"express-flash": "0.x",
"express-session": "1.x",
"passport" : "0.3.x",
"passport-local": "1.x"
}
}
We’ll then include these in our server.js file:
var express = require('express'),
cookieParser = require('cookie-parser'),
session = require('express-session'),
passport = require("passport"),
LocalStrategy = require('passport-local').Strategy,
bodyParser = require('body-parser'),
flash = require('express-flash');
Passport itself is highly configurable, and the way you add support for additional authentication schemes is by adding modules! So, we’ll include both passport and passport-local to give us all the functionality we need.
Let’s enable all these modules in our express application:
var app = express();
var session_configuration = {
secret: 'whoopity whoopity whoop whoop',
resave: false,
saveUninitialized: true,
cookie: { secure: true }
};
session_configuration.cookie.secure = false;
app.use(flash());
app.use(session(session_configuration));
app.use(cookieParser('whoopity whoopity whoop whoop'));
app.use(passport.initialize());
app.use(passport.session());
// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }))
// parse application/json
app.use(bodyParser.json())
In the above example, we turn off secure cookies because they require us to use HTTPS, which is inconvenient for development. In a production environment, we would use app.configure to ensure the right options were set for production vs development.
Finally, we make sure we support the upload of forms using both the urlencoded form format (from FORM elements on HTTP pages) as well as JSON POST data from other sources.
Laying Down the Plumbing
Next, we need to set up our app to:
Store usernames and passwords.
Configure passport-local to validate an incoming username and password against our list of users.
Serialize a user.
Deserialize a user.
Most of these are intuitive, except for perhaps the last two: passport does not serialize our entire user object to the session storage—this is potentially insecure. Instead it asks us what value it should use to store a user to session data by calling serializeUser. Then, when a new page is loaded, it will pass that value to deserializeUser, where we can pass back the full user object.
The code for the above plumbing is as follows:
var users = {
"id123456" : { id: 123456, username: "marcwan", password: "boo" },
"id1" : { id: 1, username: "admin", password: "admin" }
};
passport.use(new LocalStrategy(
function (username, password, done) {
for (userid in users) {
var user = users[userid];
if (user.username.toLowerCase() == username.toLowerCase()) {
if (user.password == password) {
return done(null, user);
}
}
}
return done(null, false, { message: 'Incorrect credentials.' });
}
));
passport.serializeUser(function (user, done) {
if (users["id" + user.id]) {
done(null, "id" + user.id);
} else {
done(new Error("CANT_SERIALIZE_INVALID_USER"));
}
});
passport.deserializeUser(function (userid, done) {
if (users[userid]) {
done(null, users[userid]);
} else {
done(new Error("CANT_FIND_USER_TO_DESERIALIZE"));
}
});
There should be no surprises in this code: our “storage” for users is just an array. We configure passport with passport.use and pass in the strategy we want to use. We’re going to use a LocalStrategy object, to which we pass our authentication function, which simply authenticates a given username/password combination against our known array. Finally, when passport wants to serialize users, we tell it to save the id#user.id string, and then we can use that to deserialize a user later in an efficient manner.
Creating a Login Form
We can make our app show a login form by adding a route to the GET /login, as follows:
app.get('/', function (req, res) {
console.log(req.flash());
res.end('<a href="/login">Login Here</a>');
});
app.get("/login", function (req, res) {
var error = req.flash("error");
var form = '<form action="/login" method="post">' +
' <div>' +
' <label>Username:</label>' +
' <input type="text" name="username"/>' +
' </div>' +
' <div>' +
' <label>Password:</label>' +
' <input type="password" name="password"/>' +
' </div>' +
' <div>' +
' <input type="submit" value="Log In"/>' +
' </div>' +
'</form>';
if (error && error.length) {
form = "<b> " + error[0] + "</b><br/>" + form;
}
res.end(form);
});
You should be able to see that submitting this form will go to POST /login. We’ll talk about what exactly req.flash is for in a bit.
Logging the User In
Next, we’ll implement the POST /login method. This is where we get passport to do all the work for us, as follows:
app.post("/login",
passport.authenticate('local', { successRedirect: '/members',
failureRedirect: '/login',
successFlash: { message: "welcome back" },
failureFlash: true })
);
Whenever we get a POST request to /login, we’ll tell passport to do the work. It’ll use the local strategy, and we’ll give it some instructions:
If the authentication succeeds, send the user to the /members page.
If the authentication fails, send the user back to the GET /login page.
We’ll set up flash messages (again, we’ll talk about this in a bit).
Under the hood, passport will use all the functions we’ve provided thus far to validate the user form. (It assumes you’re sending it username and password fields but can be configured to look for others.) It’ll then send redirects back to the browser, telling it where to go when it’s finished.
Restricting Access to a Page
Certain pages should not be accessible unless the user has logged in. To do this, we take advantage of middleware to add a function to first make sure the user can view the page, as follows:
app.get("/members", authenticatedOrNot, function (req, res) {
res.end("secret members only area!");
});
Here, we add the authenticateOrNot function:
function authenticatedOrNot(req, res, next){
if(req.isAuthenticated()){
next();
}else{
res.redirect("/login");
}
}
By calling req.isAuthenticated, we can ask passport if our session has gone through successful authentication or not. If so, we allow access to the page (by calling next and allowing the request to continue). If not, we stop processing and immediately redirect the user back to the GET /login page.
And those are the basics of how to use passport authentication in our applications. It’s extremely powerful and doesn’t require too much plumbing to be useful.
Flash Messages
Sometimes, when you’re writing an application, you want to show a message, but only once. Of course, we could write code to save the message somewhere to our session data, and then once we’ve accessed it, we delete it from said session data storage. It turns out that this is a sufficiently common thing, however, that it’s been put into its own package and called a flash message. To use these, you add express-flash to your package.json (we’re using version 0.x) and tell express to use a flash instance.
In our authentication function that we pass to the passport LocalStrategy object, you’ll see that when the request fails authentication, we call:
return done(null, false, { message: 'Incorrect credentials.' });
This message is a flash message! The next time a page request comes in, this message will be available in req.flash, which you see when we construct the login form:
var error = req.flash("error");
...
if (error && error.length) {
form = "<b> " + error[0] + "</b><br/>" + form;
}
Finally, when we have passport validate our login information in POST /login, we can set flash messages for both the success and failure cases there:
passport.authenticate('local', { successRedirect: '/members',
failureRedirect: '/login',
successFlash: { message: "welcome back" },
failureFlash: true })
Running the Sample
You can run this sample by going to the authentication/ folder in Chapter07 on GitHub. Basically, run the server and then visit localhost:8080 in the browser. Without too much code, we’ve added pretty robust authentication to our application. You’ll see this integrated into our photo-sharing application and databases in upcoming chapters.
Error Handling
Although you can handle errors individually for each request, sometimes you might like to have a global error handler for common situations. To do this, you provide a function that takes four parameters to the app.use method:
app.use(function (err, req, res, next) {
res.status(500);
res.end(JSON.stringify(err) + "
");
});
This method is placed after all the other middleware setup and routing functions so that it’s the last function that express calls in the app. You can examine the exact error in this function and decide what to return to the user. As an example, here is a version that converts any given err object or thrown Error object to appropriate JSON and returns it to the client as a server error:
var express = require('express');
var app = express();
app.get('/', function(req, res){
throw new Error("Something bad happened");
res.end('Probably will never get to this message.
');
});
app.use(function (err, req, res, next) {
res.status(500);
res.end(err + "
");
});
app.listen(8080);
Finally, Node.js allows you to specify a global application-wide error handler via the process object:
process.on('uncaughtException', function (err) {
console.log('Caught exception: ' + err);
});
It is, however, a terrible idea to use this for anything other than logging or diagnostic purposes. If this function is called, you should assume that node is in a very unhappy or unstable state and should be restarted. For those situations when you use this function, it should merely log the error and then terminate the node process:
process.on('uncaughtException', function (err) {
console.log('Caught exception: ' + err);
process.exit(-1);
});
You should use try/catch blocks around situations in which errors are possible, and for those extremely catastrophic or completely unexpected scenarios, let the process die and restart it—automatically if possible.
Summary
This is another big chapter with tons of fun new stuff. I introduced the express application framework, talked about middleware, described how to use cookies and sessions, and helped you convert some of the code in your photo-sharing application into modules. You also managed to get rid of a lot of code by using static file-serving middleware and can now add compression, authentication, and better error handling to your applications.
The photo-sharing application is shaping up quite nicely, but it still feels a bit rudimentary; you have only pictures in the albums and no great place to store any additional information you might like to add to albums or photos. With this issue in mind, in the following chapters you start looking at databases—first NoSQL (CouchDB) and then MySQL—and see how you can use them in Node. You also look at some caching solutions and continue to grow out the little app into something more useful.