In the first two parts of this book, we explored topics that are more or less "old stuff" in terms of JavaScript development. The JavaScript language has been around for quite some time, and the things we've learned so far have been explored and re-explored many times before. For some of us, the previous chapters were refreshers; for others, they're advanced information waiting to be applied.
But now we leave the comfort of the familiar and delve into a very young topic: CommonJS. Barely two years old, CommonJS has pushed JavaScript into an area it has never been successful in before: it has turned JavaScript into a powerful, general-purpose scripting language.
JavaScript has always been a language of the browser—and will remain so for quite some time. This, of course, is beneficial to the language: because the Internet has become a ubiquitous part of most people's lives, the browser—the application that helps us navigate this vast network of information—has also established itself as one of the most important applications today. And JavaScript, as the only DOM scripting language currently supported by all major browsers, benefits greatly from the continuous effort browser developers put toward making their applications faster and better.
But, at the same time, this binding of JavaScript to the browser is too limiting. JavaScript is a very powerful language that has lots of potential outside of browsers. It's a quirky language, yes, but its quirks don't make it less powerful by any significant mark.
One of the potential uses of JavaScript outside the browser is as a general-purpose scripting language. In the browser, JavaScript functions mainly as a DOM scripting language: a language for manipulating documents. Outside the browser, though, JavaScript would be a great language for handling a multitude of programming tasks: servers, text-processing, and file manipulation—just to name a few.
For JavaScript to reach this potential, though, it must lose some of its parts: no more browser-based APIs and no more DOM Trees and nodes. It needs to be lean, a language that has no other association to anything except itself. To go back to the idea of the first part of this book, JavaScript as JavaScript.
This, of course, has been done several times already. A lot of platforms have appeared over the years that tried taking JavaScript outside the browser and bringing it to the realm of the server, but mostly these didn't capture the attention of developers. This all changed in January of 2009, when for the first time, a glimmer of hope for a general-purpose JavaScript showed itself through a movement.
The CommonJS movement started in early 2009 with an effort by Kevin Dangoor. In an article called "What Server Side JavaScript Needs" (www.blueskyonmars.com/2009/01/29/what-server-side-javascript-needs/), Dangoor outlined the pitiful state of server-side JavaScript implementations at the time. He raised several points that needed to be addressed:
The lack of a standard set of libraries that can be used in any server-side JavaScript implementation.
The lack of a standard way of importing and exporting modules for use in multiple implementations.
The lack of a usable packaging system for server-side JavaScript.
To address these points, Dangoor created a new group called ServerJS, which included developers of existing server-side JavaScript implementations as well as other JavaScript developers interested in the area. The group gained quite a following, and discussions about the topics presented in the article grew into firm plans that have affected the state of JavaScript outside the browser for good.
In August of 2009—just about six months after the initial introduction of ServerJS—the group decided to rename the effort CommonJS to reflect the new direction they were taking. Instead of simply defining a set of specifications that could be used in server-side implementations, the CommonJS group wanted to make these specifications usable in any kind of JavaScript environment—the browser included.
The name CommonJS actually refers to two things: It refers to the group itself, of course, the loose organization that came into being via the ServerJS initiative. And it also refers to the set of technologies that emerged from the proposals the CommonJS group created.
The CommonJS group came out with a handful of proposals for language and API extensions, dealing with everything from modules and packages to file I/O and unit testing. As of the time of this book's writing, the group has approved and released several important specifications:
Modules (1.1)
Packages (1.0)
System Interfaces (1.0)
Unit Testing (1.0)
These four specifications—together with the dozen others that are currently being discussed—form the core set of APIs that all CommonJS-compliant engines support.
In addition to the specifications themselves, the CommonJS group is responsible for another important effect—the development of several powerful implementations that are compliant with these specifications. There are now more than a dozen implementations available, each created using the original open-source interpreter from popular browsers such as Mozilla's SpiderMonkey and Rhino engines, and Google's v8 engine. This enables a multitude of platform possibilities for JavaScript development outside the browser.
Perhaps the most important specification to come out of CommonJS is Modules. This specification is one of the first efforts of the group to be widely accepted, and it has gained support in all CommonJS engines—as well as in other platforms both inside and outside the browser.
The Modules specification—the current version is 1.1—describes a working system for creating and reusing JavaScript modules. At its center is the concept of a module, a group of related JavaScript functionalities that can be imported into CommonJS programs for reuse.
For the most part, an implementation of the Modules specification is the only requirement a platform needs in order to be called CommonJS-compliant. Any platform—from standalone JavaScript engines to fully-fledged application frameworks—can therefore be considered a CommonJS engine if it enables the use of CommonJS Modules. As such, the Modules specification plays an important role in the CommonJS ecosystem for determining which implementations can be compatible with the CommonJS philosophy.
To understand how the Modules specification—and thus the module system of all CommonJS-compliant engines—works, we'll have to implement a basic module. Let's take a module composed of some math-related functions.
var sum = function(a, b){
return a + b;
};
var increment = function(num){
return sum(num, 1);
};Here we have two simple functions, sum and increment, which we'd like to turn into a CommonJS module. In the CommonJS specification, a module is simply a file with the name of the module as the file name, followed by the file extension .js. What we want is a math module, so we'll name our file math.js:
// math.js
var sum = function(a, b){
return a + b;
};
var increment = function(num){
return sum(num, 1);
};Now we have a file called math.js, which includes our two math functions. However, we still don't have a module. To create a module, we have to define the things a module provides. A module is said to provide something if by importing the module, we can access that functionality.
In our math example, we want importers of our module to be able to access the two functions. To do this, we have to use a special object called exports:
// math.js
var sum = function(a, b){
return a + b;
};var increment = function(num){
return sum(num, 1);
};
exports.sum = sum;
exports.increment = increment;The exports object is a special global object created by a CommonJS implementation. Any properties or methods you add to this object will be available to packages that import the module. Here we added two lines at the end of our file that define the export of the sum and increment modules, thereby making these two functions importable.
Now we already have a fairly simple yet usable math module. The next step is to import it into a program that's going to use these functions. To do this, we have to use the require function:
// program.js
var math = require('math'),
console.log(math.sum(2, 3)); // 5
console.log(math.increment(303)); // 304In this example we have a very simple program stored in a file called program.js. Since our program needs both the sum and increment functions defined in our math module, we decided to import this module. We did this by invoking the require function and passing in the name of the module without the extension. This function executes the contents of the math module, and then returns the value of the exports object. Thus, the return value of the require('math') call in our example is the exports object of the math module, which is why we are able to access the sum and increment functions.
Modules are executed in their own context, and any local variables defined in a module are private to that module.
// math.js
var sum = function(a, b){
return a + b;
};
var increment = function(num){
return sum(num, 1);
};
exports.increment = increment;Here we only exported the increment function, which means that importers of this module won't be able to access the sum function. However, since modules are run in their own context, the increment function is still able to access the local function sum, which means we can use closures for our modules.
Loosely speaking, all CommonJS programs are treated as "modules" by the engine. This means that in our example, both math.js and program.js are modules, and thus any module-related API available to math.js is also available to program.js. This is an important point to remember, because it tells us we're allowed to import modules into other modules:
// sum.js
var sum = function(a, b){
return a + b;
};exports.sum = sum;
// increment.js
var sum = require('sum').sum;
var increment = function(num){
return sum(num, 1);
};
exports.increment = increment;
// program.js
var increment = require('increment').increment;
console.log(increment(302)); // 303Here we split up the math module into two modules: sum and increment. The sum module, in sum.js, is simply a declaration of the function together with an exports declaration. In increment.js, which is a separate module, we first import the sum module, then define the increment method that uses the sum function we imported. Finally, we import the increment module into the program itself. Thus, we are able to put together three separate modules into one program transparently.
Module imports using the require function are done linearly: when a module imports another module, the engine first interprets the contents of the imported module before returning the value of the exports object and continuing to interpret the rest of the importing module.
In our example, the engine will first interpret the program.js file and see that we're importing the increment module, so it will pause execution of program.js and open increment.js to start interpreting this file. In the increment.js file, it'll see that there's another call to require, so it'll again halt execution and perform the import by opening and interpreting sum.js.
After interpreting sum.js, the engine will take the value of the sum module's exports object and return to the importing module, which is increment.js. The increment.js file will then continue execution, thereby defining the increment function and exporting it. Then the engine will again take the value of the exports object and return to the main program.js file where execution will continue.
This linear execution of modules is done in order to prevent race conditions on dependencies. In our example, the increment module depends on the sum function from the sum module. Because of this dependency, the engine needs to make sure that the sum function will be available before the increment module can use it. Thus, the modules are interpreted in a linear fashion.
Those of you who are runners more than readers would probably have hit a snag in trying out the examples above. In particular, you might be getting errors about modules not being found. You probably just typed in those examples, placed them in a directory, and tried running program.js—which will lead to an error about module locations.
This error occurs because of how require works. If you notice, we didn't specify the exact location of the file to the require function, only the name of the module. How then does require know where to look for these modules?
The answer lies in a special property of the require function called paths. The require.paths property is an array of file-system locations expressed as strings. When require is invoked, it searches for the module by looking at each of these locations starting from the one indexed at 0 and moving on until it finds the particular module.
Each CommonJS engine will have a different set of default locations for require.paths, so there's not so much compatibility there. But let's say, for example, that our require.paths array looks like this:
[
'/home/mark/.commonjs',
'/home/shared/commonjs',
'/usr/local/commonjs'
]Now suppose we called require('math'). The first thing the require function will do is to inspect its paths property to determine which locations are to be searched. It takes the first location, /home/mark/.commonjs, and looks for a file called math.js in that directory. If it finds the file there, it stops the search, opens the file and interprets it. If the file doesn't exist, it will continue searching through the locations in require.paths until it finally finds the module. If all the locations have been exhausted without finding the file, require will throw an error to halt the execution of the program.
All engines allow you to add your own locations to the require.paths array. This enables us to store our modules in common locations that can be used in any CommonJS engine.
require.paths.unshift('/home/mark/.commonjs_modules'),Here we added a new location to the start of the require.paths array using the unshift method. Since paths are inspected using their indexing order, the require function will first search for an imported module in the new path before moving on to the other paths. If you want to add a new path to the end of the array, you can simply use the push method instead of unshift.
In this light, require seems to be quite absolute in its searches: in order to import a module, it needs to be stored in a location included in the require.paths array. However, this isn't always the case. The require function actually allows you to import modules relative to the current file.
Suppose our three files—program.js, increment.js, and sum.js—are stored in the /home/mark/code/example directory, which is a path that's not included in require.paths. If we run the original example without modification, our program will terminate with an error, because our modules can't be resolved. What we have to do, then, is modify our examples:
// sum.js
var sum = function(a, b){
return a + b;
};
exports.sum = sum;
// increment.js
var sum = require('./sum').sum;
var increment = function(num){
return sum(num, 1);
};
exports.increment = increment;
// program.jsvar increment = require('./increment').increment;
console.log(increment(302)); // 303Here we changed the require invocations by prefixing the module names with ./. This tells the require function that we want to search for the modules using a path that's relative to the location of the importing module. Therefore, require('./increment') will look for the increment.js file in relation to the location of the program.js file, which is at /home/mark/code/example.
Because the location is relative, we can also use the conventional directory notation. For example, say we have the following directory structure:
/example
/libs
increment.js
sum.js
program.jsNow in order for program.js to be able to import increment.js, it needs to do require('./libs/increment'). On the other hand, increment.js can use require('./sum') directly because they are both in the same location.
We saw in the first part of this book that, unlike other frameworks, MooTools does not limit itself to DOM scripting. If anything, the DOM scripting functionality in MooTools is only an extension to the real core of the framework: a powerful extension to the native JavaScript language. This makes MooTools a very good library for use with CommonJS.
Preparing MooTools for CommonJS use is as simple as building a MooTools package without browser-based extensions such as the Element type or the Request class. All other parts of the framework can be used in CommonJS, and MooTools, in fact, offers a special server-side build that can be used for CommonJS.
There is, however, a slight snag. Remember that CommonJS uses modules for much of its functionality, and to use MooTools with CommonJS engines, MooTools needs to be turned into a proper CommonJS module. The problem is that the current version of MooTools does not directly support the CommonJS module specification, which means that it doesn't use exports declarations. Instead, MooTools "exports" all its public APIs using the implicit global object, which makes exported objects like Class or Type inaccessible through require.
Thankfully, this isn't such a big problem. The structure of the MooTools library actually makes it very easy for us to add a simple importer function to make MooTools a CommonJS module. There are several available importer functions, but this one (which I've written myself) is what we'll use here:
(function(){
var $top = this,
$exports = (typeof exports === 'object') ? exports : {};
Array.each([
"MooTools",
"typeOf", "instanceOf",
"Type", "Class",
"Events", "Options", "Chain"
], function(item){ $exports[item] = $top[item]; });$exports.into = function into(globalObj){
if (globalObj && globalObj !== $top){
for (var i in $exports) {
if ($exports[i] !== into) globalObj[i] = $exports[i];
}
}
return globalObj;
};
})();To use this importer function, you have to add it to the end of the MooTools server-side build file. This adds a new function called into, which can be used to import MooTools. To use into, you simply have to call it directly and pass the global object of the CommonJS implementation you're using:
require('mootools').into(global);This example, which works in almost all major CommonJS implementations, will import MooTools and add commonly used objects such as Class or Type to the program's global object. Importing MooTools this way means that all modules—regardless of whether they import MooTools or not—will be able to use all MooTools functionality, making it easier to craft MooToolsian JavaScript programs for CommonJS.
There is, however, a simpler way that doesn't involve modifying MooTools to add an importer function: Meso.
The Meso project, which I started with the help of members of the MooTools development team and other MooTools community developers, is a cross-engine runner and library that aims to provide a cross-compatible API for using MooTools with CommonJS. Instead of targeting individual CommonJS engines, developers can target Meso to make their programs compatible with all supported implementations.
At the time of this writing, Meso supports four major engines: NodeJS, v8cgi, Flusspferd, and RingoJS, This means that all programs written using the Meso API will be able to run in all of these engines, making it easier to craft programs that can run on the user's choice of implementation.
Meso provides a simple set of classes for handling things like file system access, basic I/O, and server requests. But perhaps the most interesting feature of Meso is the runner, which not only provides a cross-engine way of running programs but also automatically imports MooTools.
Say we have the following CommonJS program:
var Person = new Class({
initialize: function(name, age){
this.name = name;
this.age = age;
},
log: function(){
return this.name + ', ' + this.age;
}
});var cassiopeia = new Person('Cassiopeia', '21'),
print(cassiopeia.log);You'll notice that we're declaring a new class in this program, but we're not importing MooTools. Because we didn't import MooTools here, the Class constructor won't be available if this program is run in a CommonJS engine, and therefore it will terminate with an error. However, if we run this on Meso, we won't get any errors because MooTools is automatically imported—which means all of those nice features we saw in the first part of this book will be usable.
Because Meso makes it quite easy to work with multiple CommonJS engines, we're going to use it instead of a specific engine for this book. You'll therefore need to install Meso on your system to be able to work through the next chapter. Because there are different installation instructions for different systems and different engines, I advise you to visit the official Meso site, http://mesojs.com, and follow the instructions there.
This chapter introduces JavaScript as a potential general-purpose scripting language—outside the browser. We learned how CommonJS and the CommonJS Modules system make this possible. We also got a glimpse of MooTools on the server side and Meso, a cross-engine toolkit for MooTools and CommonJS.
In the next chapter, we'll expand on some of the concepts we learned here and take a look at a real-world example of how MooTools can be used on the server side.
So put on that astronaut suit you stashed in your closet and get ready for launch, as we explore the space beyond the browser.