8. Databases I: NoSQL (MongoDB)
Now that you have a solid foundation for your web application and are fully set up with the express web application framework and Mustache templating, you are ready to spend a couple of chapters adding a back end to it. In these next two chapters, you look at two common ways this is done. You start in this chapter by looking at a popular NoSQL database called MongoDB, which provides a quick and easy way to serialize JSON data directly to a database. This chapter covers the basics of using it, and then you update your album handler to let you store album and photo data to the database.
I chose to work with MongoDB instead of other popular NoSQL databases—in particular CouchDB—because it’s particularly easy to use and has some wonderful querying functionality that others don’t provide so readily. Because the goal of this book is to teach you about working with Node.js, I wanted to choose the database server that was easiest to work with and would let you focus on your primary mission.
If you’re more comfortable working with MySQL or other relational databases, this chapter is still very much worth reading because you see how you move the album and photo handlers from file-based to database-backed modules. In the next chapter, where I discuss relational database systems, I provide a version of the code for MySQL as well.
Setting Up MongoDB
To use MongoDB in your Node.js applications, you need to do two things: install the database server itself and make it work with Node.
Installing MongoDB
To get started with MongoDB, you can just download the binaries from http://mongodb.com. Distributions for most platforms are in zip or .tar.gz file format, and you extract them to some location from which you’d like to run MongoDB.
In the bin/ subfolder, you should find a file called mongod, which is the base database server daemon. To run the server in a development environment, you typically just use
./mongod --dbpath /path/to/db/dir # unix
mongod --dbpath path odbdir # windows
You can create a folder like ~/src/mongodbs or something similar to hold your databases. To quickly delete them and restart your database server, just press Ctrl+C to kill the current server and then run one of the following:
rm /path/to/db/dir/* && ./mongod --dbpath /path/to/db/dir # unix
del path odbdir* # windows 1
mongod --dbpath path odbdir # windows 2
To test your installation, open another terminal window or command prompt, and run the mongo program, which runs a simple interpreter that looks a lot like the Node REPL:
Kimidori:bin marcw$ ./mongo
MongoDB shell version: 3.2.9
connecting to: test
> show dbs
local (empty)
>
For deployment to a live production environment, you should read the MongoDB documentation a bit further and learn about best practices and configuration options, including setting up replication and backups, for example.
Using Mongo DB in Node.js
After verifying the MongoDB installation, you need to hook it up to Node.js. The most popular driver being used currently is the official mongodb for Node written by the company that produces MongoDB. There is a relational-object mapping (ROM) layer called mongoose for MongoDB and Node.js that is quite popular, but you can stick with the simple driver for this book because it provides everything you want (your data needs are not terribly complicated) and lets you structure your code to be more database agnostic (as we’ll see in the next chapter).
To install the mongodb module, add the following to your package.json file:
{
"name": "MongodB-Demo",
"description": "Demonstrates Using MongoDB in Node.js",
"version": "0.0.1",
"private": true,
"dependencies": {
"async": "2.x",
"mongodb": "2.x"
}
}
You can then run npm update to get the latest version of all the drivers. If you look in the node_modules/ folder, you should see the mongodb/ subfolder there.
Structuring Your Data for MongoDB
MongoDB is quite suitable for use in your applications because it uses JSON for storage itself. Whenever you create or add things to your database, you pass around JavaScript objects—which is extremely convenient for you in Node!
If you’ve used relational databases before, you’re likely familiar with the terminology of databases, tables, and rows. In MongoDB, these elements are called databases, collections, and documents, respectively. Databases can be partitioned into collections of like objects, all of which are represented by JSON documents.
All objects in MongoDB have a unique identifier, represented by _id. It can basically be any type as long as it’s a unique value. If you do not provide one, MongoDB creates one for you: it is an instance of the class ObjectID. When printed, it returns a 24-character hexadecimal string, such as 50da80c04d403ebda7000012.
It’s All JavaScript
For the photo album application, you need to create two document types: one for albums and one for photos within albums. You do not need to use MongoDB’s autogenerated _id values because you always have unique identifying information for your documents: for albums, the album_name is always unique, and for photos, the combination of album_name and photo filename is unique.
Thus, you store your albums as follows:
{ _id: "italy2012",
name:"italy2012",
title:"Spring Festival in Italy",
date:"2012/02/15",
description:"I went to Italy for Spring Festival." }
And your photo documents look similar to the following:
{ _id: "italy2012_picture_01.jpg",
filename: "picture_01.jpg",
albumid: "italy2012",
description: "ZOMGZ it's Rome!",
date: "2012/02/15 16:20:40" }
Attempting to insert values into a collection with duplicate _id values causes MongoDB to signal an error. You use this capability to ensure that albums are unique and that image files within an album are also unique.
Data Types
For the most part, working with MongoDB in JavaScript feels entirely natural and straightforward. There are, however, a couple of scenarios in which you might run into some trouble, so it’s worth covering theme now.
As you saw back in Chapter 2, “A Closer Look at JavaScript,” JavaScript represents all numbers as 64-bit double-precision floating-point numbers. This gives you 53 bits of integer precision. But there is frequently a legitimate need for fully 64-bit integer values. When the reduced precision and approximations are unacceptable, you can use the Long class that the mongodb driver for Node has provided. It takes a string value in its constructor and lets you perform operations on 64-bit integer values. It has a number of methods, such as toString, compare, and add/subtract/multiply, to mimic all common operations you’d expect for integer values.
JavaScript also has a Binary class that, as you might expect, lets you store binary data in your documents. You can pass the constructor either a string or a Buffer object instance, and the data is stored as binary data. When you reload the document later, the value is returned to you again with some extra metadata describing how Mongo is storing it in the collection.
Finally, I want to mention the Timestamp class, which saves times to the database document but does so when you actually write the record! So you can just set a value as follows:
{ _id: "unique_idnetifier1234",
when: new TimeStamp(),
what: "Good News Everyone!" };
For a complete list of the data type helpers that the mongodb module provides, check out the documentation at github.com/mongodb/node-mongodb-native and also look at the source code in github.com/mongodb/js-bson in the directory lib/bson/.
Understanding the Basic Operations
Now it’s time to make some things happen with MongoDB and Node.js. At the top of every file that uses the mongodb module, include the following:
var MongoClient = require('mongodb').MongoClient;
For those situations in which you use extra types, such as Long or Binary, you can refer to them through the mongodb module as follows:
var mongodb = require('mongodb');
var b = new mongodb.Binary(binary_data);
var l = new mongodb.Long(number_string);
Connecting and Creating a Database
To connect to a MongoDB server, you simply refer to it with a URL pointing to the server and database you plan to use. The code to get a handle to the database (which we’ll store in db) is as follows:
var url = 'mongodb://HOSTNAME:27017/MY_DATABASE_NAME';
var db;
MongoClient.connect(url, (err, dbase) => {
if (err) return cb(err);
console.log("Connected correctly to server");
db = dbase;
});
For most of our test projects, our URL will be mongodb://localhost:27017/photosharingapp. The name of the database you want to create, or use if it already exists, is passed via the URL. It is created if it does not exist.
You do not need to create a new Db object for each incoming request to your application server, nor do you need to worry about multiple people trying to work with the database server at the same time: the mongodb module handles connection pooling and manages multiple requests at the same time for you. You can configure your mongodb connections by sending an additional object to MongoClient.connect, as follows:
MongoClient.connect(url, { w: 1, poolSize : 200 }, (err, dbase) => { ... });
MongoDB and the mongodb module you use are quite flexible with the way they handle writes and data integrity. The { w: 1 } flag you pass to the constructor (called the write concern) tells them to wait until at least one confirmed write has succeeded before calling any callbacks you provide to database operations. You can specify higher numbers for environments with replication or 0 for those cases in which you don’t care to know when the writes are finished (such as high-volume logging). Finally, "majority" may be passed instead of a number for the write concern in cases where you just want to be sure that a large enough number of your servers have confirmed the write.
You configure how many connections to maintain to the MongoDB server by adjusting the value of the poolSize option, as shown in the preceding code. For a full list of configurable options for your connection refer to the documentation on the GitHub pages for the mongodb module.
Creating Collections
As mentioned previously, collections are the MongoDB equivalent of tables from relational database systems, and you create them with a call to the collection method on the Db object. By specifying the { safe: true } option as a second parameter, you can control exactly how they work with regards to existing or nonexistent collections (see Table 8.1).
To use the { safe: true } option, you write your code as follows:
db.collection("albums", { safe: true }, (err, albums) => {
if (err) {
// album already exists or other network error.
}
// etc. ...
});
The code you will use most commonly to create or open a collection, however, is simply:
db.collection("albums", (err, albums) => {
if (err) {
console.error(err);
return;
}
// albums can now be used to do stuff ...
});
Inserting Documents into Collections
To insert a document (i.e., an object) into a collection, you use the insertOne method as follows:
var album = { _id: "italy2012",
name: "italy2012",
title: "Spring Festival in Italy",
date: "2012/02/15",
description: "I went to Italy for Spring Festival." };
albums.insertOne(album, (err, inserted_doc) => {
if (err) {
console.log("Something bad happened.");
return;
}
// continue as normal
});
You see that you are specifying your own _id field for the document. If you did not, MongoDB would provide one for you. If a document with that _id exists already, the callback is called with an error.
You can insert multiple documents at the same time by passing an array to the insertMany function:
var docs = [{ _id: "italy2012",
name: "italy2012",
title: "Spring Festival in Italy",
date: "2012/02/15",
description: "I went to Italy for Spring Festival." },
{ _id: "australia2010",
name: "australia2010",
title: "Vacation Down Under",
date: "2010/10/20",
description: "Visiting some friends in Oz!" },
{ _id: "japan2010",
name: "japan2010",
title: "Programming in Tokyo",
date: "2010/06/10",
description: "I worked in Tokyo for a while."
}];
albums.insertMany(docs, callback);
Updating Document Values
To update a document, you can call the updateOne method on a collection. The first parameter identifies a document (if multiple documents match, only the first is updated), and the second is an object description of how to modify the matching document(s). The object description comes in the form of a command and a field or fields to accompany it:
photos.updateOne({ filename: "photo_03.jpg", albumid: "japan2010" },
{ $set: { description: "NO SHINJUKU! BAD DOG!" } },
callback);
In this description, you use the command $set, which tells MongoDB to update the value of the provided field with the provided new value; here, you tell it to update the description field with a new value. There are a many different commands you can use, some of the more interesting of which are listed in Table 8.2.
As with insertOne and it’s insertMany complement, you can update multiple documents with the updateMany method, where the update applies to all documents matching the first parameter:
// Change album name of all photos in japan2010.
photos.updateOne({ albumid: "japan2010" },
{ $set: { albumid: "japan2010mkii" } },
callback);
Deleting Documents from Collections
To delete a document from a collection, you use the deleteOne method on the collection object. You can specify any set of fields to identify a document or set of documents:
photos.deleteOne({ filename: "photo_04.jpg", albumid: "japan2010" },
callback);
As with updateOne, if multiple documents match, only the first is deleted. You can skip the safe: true option and the callback if you don’t care to confirm that the delete completed:
photos.deleteOne({ filename: "photo_04.jpg", albumid: "japan2010" });
To delete more than one, you use deleteMany:
photos.deleteOne({ albumid: "japan2010" }); // delete all in this album
You can also remove all documents in a collection by simply calling deleteMany with no arguments:
photos.deleteMany(); // DANGER ZONE!!
Querying Collections
By far, the biggest reason that MongoDB is the most popular of the NoSQL database engines is that its ability to find documents in collections most closely matches the behavior of traditional SQL database queries and feels incredibly natural. All your work here is done with the find function on collections.
Find Basics
Before starting, note that the find method itself does not actually do any work; it sets up a results cursor (which is an object that can iterate over the set of rows returned from executing a query). The query is not executed and the contents of the cursor are not actually generated until you call one of the functions on it: nextObject, each, toArray, or streamRecords.
The first three operate as you’d expect: nextObject can be called to get one document, each calls a function with each resulting document, and toArray fetches all the documents and passes them as a parameter to a callback:
var cursor = albums.find();
cursor.nextObject(function(err, first_match) {});
cursor.each(function(err, document) {});
cursor.toArray(function(err, all_documents) {});
If you call the find method with no arguments, it matches all documents in the collection.
You can use the streamRecords method on the cursor to create a Stream object, which you can then use as you use other streams:
var stream = collection.find).streamRecords();
stream.on("data", function(document) {}); // why data and not readable? See text!
stream.on("end", function() {});
This is the preferred way to fetch large numbers of records because it has a lower memory footprint than methods such as toArray, which return all the documents in a single chunk. At the time of the writing of this book, the mongodb module still had not been updated to the new "readable" event model that streams in Node.js use now (since 2012 or so); it is entirely possible that this will have changed by the time you start using it. So, double-check, just to be sure. (The best place would be at the official site for the driver, mongodb.github.com/node-mongodb-native/.)
To find specific documents in your collection, you can specify the fields to match as the first parameter to the find function:
photos.find({ albumid: "italy2012" }).toArray(function (err, results));
You can use operators in find queries quite similar to those you saw previously for the update function. For example, if you had documents for people, you could find all people older than age 20 with
users.find({ age: { $gt: 20 } }).toArray(callback);
You can combine these operators using logical operations such as $and $or to give more powerful queries. For example, to return all users between the ages of 20 and 40 (inclusive), you could use
users.find({ $and: [ { age: { $gte: 20 } }, { age: { $lte: 40 } } ] });
Some more operators are shown in Table 8.3. For a full listing of them, consult the MongoDB query documentation.
Further Refining Your Finds
To implement paging and sorting on your pages, you need to be able to manipulate or otherwise modify the results of the find operations. The mongodb module lets you do this by chaining additional function calls with these effects to the find operation, before you call one of the functions to generate a cursor.
The methods you’ll use the most are skip, limit, and sort. The first indicates how many documents in the result set should be ignored before you start returning any, the second controls how many should be returned after you’ve done any skipping, and the last controls ordering and sorting—you are welcome to sort on more than one field.
So, to fetch all the photos in an album sorted ascending by date, you would write
photos.find({ albumid: "italy2012" })
.sort({ date: 1}) // 1 is asc, -1 is desc
.toArray(function (err, results));
To return the third page of these photos, assuming pages are 20 photos each, you would use
photos.find({ albumid: "italy2012" })
.sort({ date: 1}) // 1 is asc, -1 is desc
.skip(40)
.limit(20)
.toArray(function (err, results) { });
Again, anything called before the toArray function at the end is just setting up the results cursor, and the query is not executed until that last function call.
You can pass multiple fields to the sort function for sorting on multiple values:
collection.find()
.sort({ field1: -1, field2: 1 })
.toArray(callback);
Seeing it all in Action
On the GitHub page for this book, you can look in the Chapter08 folder and look for the mongo_basics/ folder there. A script called mongo_basics.js in there runs through much of what we have covered so far in this chapter. Start up a mongod server, run this script, and see what happens. You can look through the code to see how we’ve applied everything we’ve talked about so far.
Updating Your Photo Albums App
With this understanding of the basics of using MongoDB in Node.js, you can update your photo-sharing application to use a database instead of just the simple file system functions to store album and photo information. You still keep the actual image files on your hard disk, but in a real production environment, you would put them on some sort of storage server or content delivery network (CDN).
Writing the Low-Level Operations
You begin by adding a new folder named data/ to your application. You put low-level operations to the back end in this folder. You also create a file called backend_helpers.js that will contain some functions to help you work with and generate errors, validate parameters, and do some other common back-end operations. These tasks are all pretty simple, and you can view the source code on GitHub.
Creating a Configuration File
In the application root folder, create a new file called local.config.json, which contains the following JavaScript:
{
"config" : {
"db_config": {
"host_url": "mongodb://localhost:27017/photosharingapp"
},
"static_content": "../static/"
}
}
Including this file means that all of the configuration options are handily located in one place, and you can change it trivially without having to rummage through code looking for things.
Getting Your Databases and Collections
Next, create a file called db.js in data/. In this file, you create the database connection and collections you use for your photo application. In this file, you also create the connection to the PhotoSharingApp database using the information in the local.config.json file:
var MongoClient = require('mongodb').MongoClient,
async = require('async'),
local = require("../local.config.json");
// We'll keep this private and not share it with anybody.
var db;
// Before the app can start, call this method. If it fails, don't start!
exports.init = function (callback) {
async.waterfall([
// 1. open database connection
function (cb) {
console.log("
** 1. open db");
var url = local.config.db_config.host_url;
MongoClient.connect(url, (err, dbase) => {
if (err) return cb(err);
console.log("** Connected to server");
db = dbase;
cb(null);
});
},
Next we’ll get the collections for albums and photos within those albums using the collection method. You do this by adding the following code to the async.waterfall we started above in the init function:
// 2. create collections for our albums and photos. if
// they already exist, then we're good.
function (cb) {
console.log("** 2. create albums and photos collections.");
db.collection("albums", cb);
},
function (albums_coll, cb) {
exports.albums = albums_coll;
db.collection("photos", cb);
},
function (photos_coll, cb) {
exports.photos = photos_coll;
cb(null);
}
], callback);
};
exports.albums = null;
exports.photos = null;
You can see that albums and photos are now exported objects in db.js, so people can fetch them whenever they need them by writing
var db = require('./db.js');
var albums = db.albums;
Finally, you need to make sure your db.init function is called before the application starts, so replace the call in server.js:
app.listen(8080);
with
db.init( (err, results) => {
if (err) {
console.error("** FATAL ERROR ON STARTUP: ");
console.error(err);
process.exit(-1);
}
console.log("** Database initialized, listening on port 8080");
app.listen(8080);
});
Creating an Album
We’ll now create a file called album.js in data/. This file will contain all the backend primitive operations you can perform on an album using mongodb.
The code to create a new album is thus:
exports.create_album = function (data, callback) {
var final_album;
var write_succeeded = false;
async.waterfall([
function (cb) {
try {
backhelp.verify(data,
[ "name", "title", "date", "description" ]);
if (!backhelp.valid_filename(data.name))
throw invalid_album_name();
} catch (e) {
cb(e);
}
cb(null, data);
},
// create the album in mongo.
function (album_data, cb) {
var write = JSON.parse(JSON.stringify(album_data));
write._id = album_data.name;
db.albums.insert(write, { w: 1, safe: true }, cb);
},
// make sure the folder exists in our static folder.
function (new_album, cb) {
write_succeeded = true; // inserted the album.
final_album = new_album[0];
fs.mkdir(local.config.static_content
+ "albums/" + data.name, cb);
}
],
function (err, results) {
if (err) {
if (write_succeeded)
db.albums.remove({ _id: data.name }, function () {});
if (err instanceof Error && err.code == 11000)
callback(backhelp.album_already_exists());
else if (err instanceof Error && err.errno != undefined)
callback(backhelp.file_error(err));
else
callback(err);
} else {
callback(err, err ? null : final_album);
}
});
};
Although the async module makes the code a bit “longer” and spread out, I hope you can already see how much it cleans things up. All the asynchronous operations are expressed as a sequence of things that need to be done; async takes care of all the messy details for you!
You do one trick here in the code to clone an object:
var write = JSON.parse(JSON.stringify(album_data));
It turns out that quickly serializing and deserializing an object is one of the quickest ways to clone it in JavaScript. We clone the object in the code above so that we do not modify an object that “isn’t ours” per se. It can be considered rude (or downright buggy) to modify objects passed in to our function, so we have just quickly cloned it before adding the _id field. Note that backhelp is simply the backend (in the data/ folder) version of the helper functions we’ve seen before in helpers.js.
Finding Albums
Loading an album given its name is quite an easy operation:
exports.album_by_name = function (name, callback) {
db.albums.find({ _id: name }).toArray((err, results) => {
if (err) {
callback(err);
return;
}
if (results.length == 0) {
callback(null, null);
} else if (results.length == 1) {
callback(null, results[0]);
} else {
console.error("More than one album named: " + name);
console.error(results);
callback(backutils.db_error());
}
});
};
You spend more time on error handling and validation than you do actually finding the album!
Listing All Albums
Similarly, listing all the albums is quite easy:
exports.all_albums = function (sort_field, sort_desc, skip, count, callback) {
var sort = {};
sort[sort_field] = sort_desc ? -1 : 1;
db.albums.find()
.sort(sort)
.limit(count)
.skip(skip)
.toArray(callback);
};
exports.photos_for_album = function (album_name, pn, ps, callback) {
var sort = { date: -1 };
db.photos.find({ albumid: album_name })
.skip(pn)
.limit(ps)
.sort("date")
.toArray(callback);
};
Adding a Photo to an Album
Indeed, the only other semicomplicated operation you need to perform is adding a photo to an album, and that is a bit more work because you have to add the step to copy the uploaded temporary file to its final place in the static/albums/ folder:
exports.add_photo = function (photo_data, path_to_photo, callback) {
var final_photo;
var base_fn = path.basename(path_to_photo).toLowerCase();
async.waterfall([
function (cb) {
try {
backhelp.verify(photo_data,
[ "albumid", "description", "date" ]);
photo_data.filename = base_fn;
if (!backhelp.valid_filename(photo_data.albumid))
throw invalid_album_name();
} catch (e) {
cb(e);
}
cb(null, photo_data);
},
// add the photo to the collection
function (pd, cb) {
pd._id = pd.albumid + "_" + pd.filename;
db.photos.insert(pd, { w: 1, safe: true }, cb);
},
// now copy the temp file to static content
function (new_photo, cb) {
final_photo = new_photo[0];
var save_path = local.config.static_content + "albums/"
+ photo_data.albumid + "/" + base_fn;
backhelp.file_copy(path_to_photo, save_path, true, cb);
}
],
function (err, results) {
if (err && err instanceof Error && err.errno != undefined)
callback(backhelp.file_error(err));
else
callback(err, err ? null : final_photo);
});
};
Modifying the API for the JSON Server
Next, you add the following two new API functions to the JSON server to facilitate creating albums and adding photos to them:
app.put('/v1/albums.json', album_hdlr.create_album);
app.put('/v1/albums/:album_name/photos.json', album_hdlr.add_photo_to_album);
The nice thing about express is that it makes this process that simple. When you add these two lines, the API is expanded with new functionality, and now you just need to update the album handler to support these new features.
Because the API now supports putting data, including files and POST bodies, you need to add some more middleware to support this capability, and we need to be sure we’re including all the correct files. So, the top of our server.js file will now look as follows:
var express = require('express'),
bodyParser = require('body-parser'),
morgan = require('morgan'),
multer = require('multer');
var db = require('./data/db.js'),
album_hdlr = require('./handlers/albums.js'),
page_hdlr = require('./handlers/pages.js'),
helpers = require('./handlers/helpers.js');
var app = express();
app.use(express.static(__dirname + "/../static"));
app.use(morgan('dev'));
// Parse application/x-www-form-urlencoded & JSON and set up file uploads.
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())
var upload = multer({ dest: "uploads/" });
We are going to use multer again to handle our file uploads and body-parser to handle POST data, and we’ll set up morgan for logging and make sure to include express static file handling.
We’ll also be sure to include all the modules that we’ve written that we’ll need to coordinate things. Although db.js contains mostly backend functions we shouldn’t be calling from server.js, we do need to initialize the database, so we include it here.
Updating Your Handlers
Now that you have the database primitives for your album and photo operations, you need to modify the album handler to use them instead of file system operations.
Some Helpful Classes
Begin by creating a couple of helpful little classes. The one for photos is as follows:
function Photo (photo_data) {
this.filename = photo_data.filename;
this.date = photo_data.date;
this.albumid = photo_data.albumid;
this.description = photo_data.description;
this._id = photo_data._id;
}
Photo.prototype._id = null;
Photo.prototype.filename = null;
Photo.prototype.date = null;
Photo.prototype.albumid = null;
Photo.prototype.description = null;
Photo.prototype.response_obj = function() {
return {
filename: this.filename,
date: this.date,
albumid: this.albumid,
description: this.description
};
};
The only really interesting function here is response_obj. You use it because the Photo class theoretically holds everything you could want to know about a photo, but when you pass it back in JSON to the caller of an API, there might be some data you don’t want to include in that response object. Consider a User object; you would certainly want to scrub out passwords and other sensitive data.
A basic version of an Album object would also look as follows:
function Album (album_data) {
this.name = album_data.name;
this.date = album_data.date;
this.title = album_data.title;
this.description = album_data.description;
this._id = album_data._id;
}
Album.prototype._id = null;
Album.prototype.name = null;
Album.prototype.date = null;
Album.prototype.title = null;
Album.prototype.description = null;
Album.prototype.response_obj = function () {
return { name: this.name,
date: this.date,
title: this.title,
description: this.description };
};
Now, look at how the handler class is reorganized to use the low-level album operations you wrote in the previous section.
Creating an Album
Again, you almost write more code checking for errors than you do performing the operation. This is the nature of good coding. Too many books and tutorials skip over these things, and it’s one of the reasons there’s so much bad code in the world today!
var album_data = require('../data/album.js');
// ... etc. ...
exports.create_album = function (req, res) {
async.waterfall([
// make sure the albumid is valid
function (cb) {
if (!req.body || !req.body.name) {
cb(helpers.no_such_album());
return;
}
// TODO: we should add some code to make sure the album
// doesn't already exist!
cb(null);
},
function (cb) {
album_data.create_album(req.body, cb);
}
],
function (err, results) {
if (err) {
helpers.send_failure(res, helpers.http_code_for_error(err), err);
} else {
var a = new Album(results);
helpers.send_success(res, {album: a.response_obj() });
}
});
};
Loading an Album by Name
Once again, error checking and handling make up 90 percent of your work. I highlighted the call here to the back end that actually fetch the album:
exports.album_by_name = function (req, res) {
async.waterfall([
// get the album
function (cb) {
if (!req.params || !req.params.album_name)
cb(helpers.no_such_album());
else
album_data.album_by_name(req.params.album_name, cb);
}
],
function (err, results) {
if (err) {
helpers.send_failure(res, helpers.http_code_for_error(err), err);
} else if (!results) {
helpers.send_failure(res,
helpers.http_code_for_error(err),
helpers.no_such_album());
} else {
var a = new Album(album_data);
helpers.send_success(res, { album: a.response_obj() });
}
});
};
Listing All Albums
Here, you fetch only 25 albums at a time so you will not have overly complicated pages. You could make these configurable via query parameters if you wanted.
exports.list_all = function (req, res) {
album_data.all_albums("date", true, 0, 25, function (err, results) {
if (err) {
helpers.send_failure(res, err);
} else {
var out = [];
if (results) {
for (var i = 0; i < results.length; i++) {
out.push(new Album(results[i]).response_obj());
}
}
helpers.send_success(res, { albums: out });
}
});
};
This model of separating the handlers and database code seems as though it creates a bit more work (and it does a little bit), but it has the huge advantage of giving you a very flexible back end. In the next chapter you see that you can switch the data storage for albums and photos to another database system without touching the album handler at all! You only need to modify the classes in the data/ folder.
Getting All Photos for an Album
The code to view photos in an album is shown in Listing 8.1. It involves two new methods, exports.photos_for_album and a new function to the Album object, photos. Most of the complexity of these functions comes from handling paging and slicing up the output array of photos.
Listing 8.1 Getting all the Photos in an Album
Album.prototype.photos = function (pn, ps, callback) {
if (this.album_photos != undefined) {
callback(null, this.album_photos);
return;
}
album_data.photos_for_album(
this.name,
pn, ps,
function (err, results) {
if (err) {
callback(err);
return;
}
var out = [];
for (var i = 0; i < results.length; i++) {
out.push(new Photo(results[i]));
}
this.album_photos = out;
callback(null, this.album_photos);
}
);
};
exports.photos_for_album = function(req, res) {
var page_num = req.query.page ? req.query.page : 0;
var page_size = req.query.page_size ? req.query.page_size : 1000;
page_num = parseInt(page_num);
page_size = parseInt(page_size);
if (isNaN(page_num)) page_num = 0;
if (isNaN(page_size)) page_size = 1000;
var album;
async.waterfall([
function (cb) {
// first get the album.
if (!req.params || !req.params.album_name)
cb(helpers.no_such_album());
else
album_data.album_by_name(req.params.album_name, cb);
},
function (album_data, cb) {
if (!album_data) {
cb(helpers.no_such_album());
return;
}
album = new Album(album_data);
album.photos(page_num, page_size, cb);
},
function (photos, cb) {
var out = [];
for (var i = 0; i < photos.length; i++) {
out.push(photos[i].response_obj());
}
cb(null, out);
}
],
function (err, results) {
if (err) {
helpers.send_failure(res, err);
return;
}
if (!results) results = [];
var out = { photos: results,
album_data: album.response_obj() };
helpers.send_success(res, out);
});
};
Adding a Photo
Finally, you can write the API to add photos; it is shown in Listing 8.2. This API also involves adding a new method to the Album object. I’ve highlighted the code that handles the incoming file upload, as it’s how multer tells us what the user sent.
Listing 8.2 Adding Photos Using the API
Album.prototype.add_photo = function (data, path, callback) {
album_data.add_photo(data, path, function (err, photo_data) {
if (err)
callback(err);
else {
var p = new Photo(photo_data);
if (this.all_photos)
this.all_photos.push(p);
else
this.app_photos = [ p ];
callback(null, p);
}
});
};
exports.add_photo_to_album = function (req, res) {
var album;
async.waterfall([
// make sure we have everything we need.
function (cb) {
if (!req.body)
cb(helpers.missing_data("POST data"));
else if (!req.file)
cb(helpers.missing_data("a file"));
else if (!helpers.is_image(req.file.originalname))
cb(helpers.not_image());
else
// get the album
album_data.album_by_name(req.params.album_name, cb);
},
function (album_data, cb) {
if (!album_data) {
cb(helpers.no_such_album());
return;
}
album = new Album(album_data);
req.body.filename = req.file.originalname;
album.add_photo(req.body, req.file.path, cb);
}
],
function (err, p) {
if (err) {
helpers.send_failure(res, helpers.http_code_for_error(err), err);
return;
}
var out = { photo: p.response_obj(),
album_data: album.response_obj() };
helpers.send_success(res, out);
});
};
Adding Some New Pages to the Application
The JSON server is now completely modified to use MongoDB for all its album and photo storage work. What you don’t have yet, however, are some pages to let you create new albums or add new photos to albums via the web interface. Let’s fix that now.
Defining the URLs for the Pages
Here, you place the two new pages you want to create in /pages/admin/add_album and /pages/admin/add_photo. Fortunately, you don’t need to modify the URL handlers in the express app for this at all.
Creating an Album
Don’t forget that for each page in the Mustache-templated site, you need two files:
A JavaScript bootstrapper
A template HTML file
The code to bootstrap the add album page is straightforward; it doesn’t even need to load any JSON from the server, only the template. It is shown in Listing 8.3.
Listing 8.3 admin_add_album.js
$(function(){
var tmpl, // Main template HTML
tdata = {}; // JSON data object that feeds the template
// Initialize page
var initPage = function() {
// Load the HTML template
$.get("/templates/admin_add_album.html", function (d){
tmpl = d;
});
// When AJAX calls are complete parse the template
// replacing mustache tags with vars
$(document).ajaxStop(function () {
var renderedPage = Mustache.to_html( tmpl, tdata );
$("body").html( renderedPage );
})
}();
});
The code for the HTML page to add the album is a bit more complicated, because you need to write some JavaScript to do the form submission via Ajax. This code is shown in Listing 8.4. The trickery you see with the dateString variable ensures that dates are always in the format yyyy/mm/dd, and not sometimes yyyy/m/d.
Listing 8.4 admin_add_album.html
<form name="create_album" id="create_album"
enctype="multipart/form-data"
method="PUT"
action="/v1/albums.json">
<h2> Create New Album: </h2>
<dl>
<dt>Album Name:</dt>
<dd><input type='text' name='name' id="name" size='30'/></dd>
<dt>Title::</dt>
<dd><input id="photo_file" type="text" name="title" size="30"/></dd>
<dt>Description:</dt>
<dd><textarea rows="5" cols="30" name="description"></textarea></dd>
</dl>
<input type="hidden" id="date" name="date" value=""/>
</form>
<input type="button" id="submit_button" value="Upload"/>
<script type="text/javascript">
$("input#submit_button").click(function (e) {
var m = new Date();
var dateString =
m.getUTCFullYear() +"/"+
("0" + (m.getUTCMonth()+1)).slice(-2) +"/"+
("0" + m.getUTCDate()).slice(-2) + " " +
("0" + m.getUTCHours()).slice(-2) + ":" +
("0" + m.getUTCMinutes()).slice(-2) + ":" +
("0" + m.getUTCSeconds()).slice(-2);
$("input#date").val(dateString);
var json = "{ "name": "" + $("input#name").val()
+ "", "date": "" + $("input#date").val()
+ "", "title": "" + $("input#title").val()
+ "", "description": "" + $("textarea#description").val()
+ "" }";
$.ajax({
type: "PUT",
url: "/v1/albums.json",
contentType: 'application/json', // request payload type
"content-type": "application/json", // what we want back
data: json,
success: function (resp) {
alert("Success! Going to album now");
window.location = "/pages/album/" + $("input#name").val();
}
});
});
</script>
Adding Photo to Album
To add a photo to the album, you have to write a bit more complicated code. In the bootstrapper, you need a list of all the albums so that the user can select to which album the photo should be added. This code is shown in Listing 8.5.
Listing 8.5 admin_add_photo.js
$(function (){
var tmpl, // Main template HTML
tdata = {}; // JSON data object that feeds the template
// Initialize page
var initPage = function () {
// Load the HTML template
$.get("/templates/admin_add_photos.html", function (d){
tmpl = d;
});
// Retrieve the server data and then initialize the page
$.getJSON("/v1/albums.json", function (d) {
$.extend(tdata, d.data);
});
// When AJAX calls are complete parse the template
// replacing mustache tags with vars
$(document).ajaxStop(function () {
var renderedPage = Mustache.to_html( tmpl, tdata );
$("body").html( renderedPage );
})
}();
});
And finally, I leave you to look in the create_album/ folder in the GitHub source for Chapter 8 for the source of the HTML page that shows the form and performs the actual uploads to the server (static/templates/admin_add_photo.html). The big thing this file does is use the FormData extension to the XmlHttpRequest object to allow Ajax file uploads, as shown here:
$("input#submit_button").click(function (e) {
var m = new Date();
var dateString = /* process m -- see GitHub */
$("input#date").val(dateString);
var oOutput = document.getElementById("output");
var oData = new FormData(document.forms.namedItem("add_photo"));
var oReq = new XMLHttpRequest();
var url = "/v1/albums/" + $("#albumid").val() + "/photos.json";
oReq.open("PUT", url, true);
oReq.onload = function (oEvent) {
if (oReq.status == 200) {
oOutput.innerHTML = "
Uploaded! Continue adding or <a href='/pages/album/"
+ $("#albumid").val() + "'>View Album</a>";
} else {
oOutput.innerHTML = "
Error " + oReq.status + " occurred uploading your file.<br />";
}
};
oReq.send(oData);
});
FormData is powerful and awesome but is not supported in Internet Explorer versions before 10. Firefox, Chrome, and Safari have all supported it for quite a while. If you must support older IE browsers, you need to look at other methods for uploading files, such as using Flash or otherwise regular old HTML forms.
Recapping the App Structure
The application has gotten a bit more complicated; it is worth spending a few seconds again covering exactly how you’ve structured it. You’ve moved all static content into the static/ folder and the code into the app/ folder, so you now have the following basic layout:
The static/ folder contains the following subfolders:
albums/—Contents our albums and their image files
content/—Contains stylesheets and JavaScript bootstrapping files needed to render the page templates
templates/—The HTML templates for rendering pages in the client browser
In the app/ folder, you have:
./—Contains the core server scripts and package.json files
data/—All code and classes related to working with the backend data store
handlers/—Contains the classes that are responsible for handling incoming requests
All of the versions of your application from this point on will use this structure.
Summary
You now have not only a fully updated version of the photo album application that uses MongoDB as its data store for albums and photos but also a couple of more interesting pages in your web application for creating albums and uploading photos to the server.
The only problem is that anybody can view and use these pages and APIs to manipulate albums and photos. So you next need to focus your attention on adding users and requiring that the users be logged in before making these changes.