Section 6: Extending Mongrel
6.4 Distributing Your Plugins as RubyGems
It is surprisingly trivial to extend Mongrel to suit your own needs. You can write your own commands, request handlers, filters, and quickly configure them as needed. Unlike other systems, though, you don’t have to know anything mentioned in this section to use Mongrel. All of this information is “behind the dirty curtain” and only for the very adventurous types who need it.
It’s possible to have your first handler up and running in about ten minutes, and fairly extensive plugins can be written within hours or days. It’s not as nice as Rails, Nitro, or Camping, where you can create whole applications in days. When you extend Mongrel, you’re working at the bare-metal death’s edge and have to do more work to keep things straight. The advantage is that you have full control over what happens and you can squeeze even more speed out of the beast.
6.1. Mongrel’s Architecture
What makes Mongrel so extensible is its simplistic architecture that gets the job of cooking and shuttling HTTP requests around the system done. Mongrel would probably make the Enterprisey folks scream since it is obviously too simple to scale. Ironically this simplicity is exactly why Mongrel is fast and scales, but it’s also your secret weapon when extending and embedding to your own crazy ideas.
Zed Sez
Simpler always wins when you are writing software. I didn’t say “simplistic” but “simpler,” meaning when compared to another solution. I find that too often programmers get too caught up in building grand palaces of crystal and gold with incredibly intricate inner workings. They love impressing their friends with the equivalent of the “SUV” of software architectures and the more boxes, lines, and color in the diagram the better. Even better if it has to go on multiple walls.
What we have now is a crisis where the coders of the world value complexity and intricacy over simplicity and directness. They would rather build a “Rube Goldberg Architecture”[9] full of indirect puzzles that interact in a chain to accomplish nearly nothing. They then wonder why their software is buggy, crashes, has security holes, and requires a coterie of 10 developers to gather before it works.
I’ve always had a different aesthetic sense when I write my software. I value simplicity and directness and try to write software that follows this approach. I jokingly call it the Shibumi School of Software Structure. All I do is apply this rule: When given two possible designs with equal end results, pick the simpler one. I then ruthlessly strip the solution down to its finest elements, but no more.
Keep this in mind when you’ve got an architecture or solution designed that you feel is fantastic. Are you actually impressed with the solution or its complexity? Are you sure there’s not a simpler way to do the same thing, even if it seems “ugly”? Is it possible that all the people who told you this is how software is designed are wrong?
These are the types of questions developers should be asking. Until they do, we’re just going to get larger and larger architectures that don’t really do anything. Rube would be proud.
Mongrel’s architecture is composed of the following key players:
HttpServer The core center of Mongrel that processes incoming HTTP requests and coordinates all the other classes to complete them.
HttpRequest Created by the HttpServer after the incoming HTTP headers and body are processed, it contains all the information your HttpHandler needs to do its work.
HttpResponse This is crafted and then handed to the HttpHandler as the way the handler builds its response to the client.
HttpHandler The class responsible for processing cooked HttpRequest and HttpResponse objects in order to complete a request. This is where you put your logic.
URIClassifier You don’t really need to use this, but it is responsible for mapping request paths to chains of HttpHandlers for processing. This is what your handler gets registered with.
That makes up the main classes being used. Yes, that’s all of them. There are a few others that help setting up new handlers, help package them as plugins, etc. but you can get by with just these five, and actually you don’t have to know how HttpServer and URIClassifier work.
How all of these classes work together is best illustrated in Figure 6.1.
Figure 6.1. Mongrel Request Processing
The only thing missing from this diagram is that HttpServer does this for chains of handlers that are registered at different URI paths. To make sure you understand the diagram, we’ll go through a typical request.
- HttpServer accepts a new client and parses their headers. At this point the server will reject clients if their headers do not parse.[10]
- The results of parsing are passed to a new HttpRequest, which becomes the HttpRequest#params variable used by HttpHandler to process forms and queries.
- HttpRequest continues processing the client in order to build up the body of the request. The body is always stored in an IO object, but it might be a temporary file or a StringIO depending on the size of the body.
- Once the HttpRequest is constructed, it and a HttpResponse object are passed to the first registered HttpHandler to do the processing.
- The HttpHandler#process does its magic and builds the response using the HttpResponse#start method call to pass back headers and the body content. Remember that you are at bare metal so you must explicitly specify most headers manually.
- Finally, when the HttpHandler#process method is done, the server checks the HttpRequest it passed and determines if the handler chain is done. If not, then it loads the next handler and continues until there are no more.
The only missing piece is how the HttpServer knows which handler gets which requests. As mentioned in the list of important classes, URIClassifier keeps a mapping of all paths and the HttpHandler chains that are responsible for processing them. HttpServer just looks them up in URIClassifier like a hash map and it’s done. You actually won’t work with this directly except for simple registration calls through Configurator#uri.
6.1.1. Documentation
Now that you understand Mongrel’s deceptively simple architecture, you should take a quick break to browse through the Mongrel Web site’s RDoc documentation. Much effort was put into documenting every class in Mongrel so that developers can be self-sufficient. You can find Mongrel’s documentation at the Mongrel RubyForge RDoc pages (http://mongrel.rubyforge.org/rdoc/).
6.2. Handlers
Let’s jump right into things and make a very simple handler that only reflects its parameters back to the Web browser. Put this code into a file named mongrel.conf:
Simple enough. We just say ParamDumper inherits HttpHandler, create a process method, and use the HttpResponse object to dump the HttpRequest#params variable via inspect. The last line actually calls Configurator#uri, but there’s a bit of voodoo going on that we’ll explain later. For now just understand that’s how you attach a handler to a URI path. Notice also that we set the “Content-Type” to “text/plain” using the head variable.
Loading this for now will be as simple as:
mongrel_rails start -e production -S mongrel.conf
The -S option (which we first covered in Section 4) basically means “run this file as a configure script.” A configure script is a Ruby script, but it is run within the context of an already configured Configurator object. This means that all the functions in Configurator like uri can be called as if they were global functions. The Configurator#uri function is the one you’ll be calling the most.
When you get this running and it is working you should be able to go to http://localhost:3000/dumped with a browser and it’ll spew the results to a text page. That’s it, you’ve just written your first HttpHandler.
6.3. Filters
Sometimes you don’t really want to handle a request, but you want to be able to modify the results coming in or going out of the server dynamically. Mongrel’s HttpHandler classes can do this since they by default act in a chain. In this situation you are ostensibly writing a “filter,” which is really just a handler that modifies its results.
This brings up another point: handlers don’t have to do any processing. It’s perfectly legal for them to make a decision and do nothing, allowing the remaining handlers do their thing instead.
For our first filter we’re going to write a class that removes any HTTP_USER_AGENT headers coming in, and then another filter that deflate compresses (deflates) its output. First the UserAgentRemover:
Pretty simple, we just delete it from the params hash and log a simple message. If you need to do simple logging then use STDERR since this is written to the log/mongrel.log file or the console and isn’t hijacked by the framework. If you need to do serious logging then use Ruby’s Logger.
The above code assumes that you put this in the mongrel.conf after the first uri call. This is why this block of code as a :in_front parameter, which tells the Configurator that you want this handler moved to the front of the chain.
Run your mongrel_rails again and hit the /dumped URI to see that HTTP_USER_AGENT is actually removed.
This ability to modify requests as they come in with simple handler chains means that you can perform security checks, clean up requests, do file processing, and other tasks prior to Rails or another framework getting the results. We’ll now write another handler that does nearly the same thing but it takes results and translates them into deflate form. Add this next block to your mongrel.conf file:
Obviously this handler is more involved, but it’s a good example of doing something useful with HttpHandlers as filters. Let’s walk through it:
- It shows how to set up some default options and write your own initialize. HttpHandler is very thin and you can modify the initialize method however you want. In this case we allow a Hash parameter style and let the user configure the handler.
- We then peel out the encodings the client accepts so we can see if they accept “deflate.”
- If they accept it then we set the “Content-Encoding” to “deflate” and we deflate compress the body using Ruby’s Zlib::Deflate API. Notice that we have to do some shuffling of the response body. If we were to gzip the response directly, and the previous handler had attached a file to send, then we’d be gzip their files silently.
- Instead we deflate their file into a StringIO, detach it and send it on, closing the original.
This filter isn’t put on the handler chain with :in_front so it will happen after ParamDumper. If we were to draw a picture of this setup it’d look like Figure 6.2, even though in the mongrel.conf we added them in a different order. This is necessary since, if you need to put a filter in front of say RailsHandler, then :in_front lets you do that.
Figure 6.2. Handler Chain Processing
6.4. Distributing Your Plugins as RubyGems
While it is easy to type your handlers into a -S mongrel.conf, this isn’t an easy way to distribute your handlers and it’ll break down if you need extras to go along with them. The preferred way to package up a Mongrel plugin and distribute it to people is to use the GemPlugin system.
GemPlugin is a dynamic plugin loading system based entirely on RubyGems. It’s very simple to use and comes with a full project generator to get you started. GemPlugin makes it possible for people to extend Mongrel without including their source in Mongrel, and to distribute them in a way that’s familiar to most Ruby users.
Explaining how GemPlugin works probably isn’t going to be helpful since it digs into the RubyGems API. Instead, we’ll do a quick demonstration of the PFM[11] in action. Take your install of Mongrel and make sure you remove the mongrel_cluster gem:
Notice the list of commands when you run the mongrel_rails command? Add the mongrel_cluster gem back to your system and run mongrel_rails again and you’ll see this list of commands:
Now you get all these new commands like magic. No, like PFM. How’s that work? The simplest explanation is that GemPlugin goes through the list of gems installed in your system and tries to find gems that depend on the mongrel gem. When it does, it requires the gem and runs an init.rb file if present.
With this simple system, Mongrel will then autoload any plugins you install as gems. We’ll go through two examples of doing a GemPlugin in the next few sections.
6.4.1. Using Rubyforge to Share
When you’ve crafted a nice GemPlugin and you want to share it with the world, you should pop over to http://www.rubyforge.org/ and register a project. When you do this you can upload your .gem file to their file service and then the RubyGems mirrors will pick it up.
6.5. Commands
Our first example of using GemPlugins will be to write a new command for Mongrel to make it quack like a duck. Matt has an affinity for quacks, so it seemed appropriate. Mongrel’s commands are implemented as GemPlugins so they can be added easily without modifying Mongrel’s source directly.
6.5.1. Generate the Project
GemPlugin comes with a command called gpgen, which generates a skeleton project ready for you to use. You run it like you would the rails command to start a project:
$ gpgen duck
This will print out the files being created, most of which you won’t have to edit. The important files are:
duck/COPYING Your license or copying restrictions.
duck/lib Where you store your source for the plugin to use.
duck/Rakefile Builds your stuff.
duck/resources A place to put files your plugin might need.
duck/lib/duck/init.rb Required to initialize your plugin.
duck/resources/defaults.yaml Default configuration options.
We’ll be skipping the defaults.yaml and resources for now. They let you package additional configuration information with your gem plugin, which GemPlugin will also load. You should refer to the GemPlugin RDoc[12] for more information.
6.5.2. Set Up the Project Files
In order to set up our duck command, we need to tweak the Rakefile and put some code in the init.rb, then simply install the gem and go. First the Rakefile needs some lovin’:
This will configure the gem task for Rake so that it is properly attributed, has the right dependencies for GemPlugin (both gem_plugin and mongrel), and includes any resources (we have none). Try to build it to make sure at least this much was done.
$ rake
$ gem install pkg/quack-0.1.gem
You should see some really boring build stuff and then in the pkg directory is the gem. Install and uninstall it to make sure it works.
If you were serious about distributing this gem you’d also want to update the COPYING and LICENSE files (not sure why this needs both) and add any other dependencies you need.
6.5.3. Write the Initialize File
With our build working, we now just need to toss some code into the init.rb file to get things in motion. This is where we take a slight diversion into “Why the Lucky Stiff” land. Mr. Stiff wrote a really cool article about hijacking contexts. Using this, GemPlugin can have a “second name space” that you register your plugins with outside of the usual Ruby Module system. This is handy for finding plugins by a sane category system without trolling through all the registered classes.
The best way to show you how this works is to just show you the Duck command’s code and then explain it:
We start off by simply requiring the libraries we need, and then we declare the Duck class. See that odd syntax in that GemPlugin::Plugin “/commands” bit of code? That’s the weird “register this class under /commands” way of making a GemPlugin. Using this class declaration style (which is not standard Ruby in any way), Mongrel is able to ask GemPlugin to give it a list of all classes registered under “/commands” and then run them as needed based on names. It’s quite voodoo but it does have a great usage in this case.
The next step is to include the Mongrel::Command::Base module to give your class all the features of a Mongrel command. This hooks up a bunch of extra code, so you should go read the RDoc for this in the Mongrel documentation.[13]
Once we’ve got this basic setup going we just need to write three functions: configure, validate, and run. They are run in this order and do basically what they say. The contents of configure are almost always setting up options, but you can do other stuff that needs to work before validation.
The validate function then uses a set of handy valid? methods to check for conditions, reports errors and aborts if anything doesn’t match. The return value of this function should be @validate (it just should be, don’t ask why).
With that all over with, the run command should be set up, have all its settings validated, and should be ready to go. As you can see, we print out a series of annoying “QUACK!” messages in honor of Matt.
6.5.4. Install and Play
This part is very simple; you just need to run rake again and install:
$ rake
$ gem install pkg/quack-0.1.gem
Now when you run mongrel_rails, you should see your fancy duck command, and you can then do:
$ mongrel_rails duck -a 10
Wow, isn’t that annoying? Quack, quack, quack all day long.
6.6. Handlers as GemPlugins
Handlers can also be GemPlugins, and they are the preferred way to distribute your handlers. If we take our UserAgentRemover handler and turn it into a GemPlugin, the code would be:
The only change made to the code was to do the usual GemPlugin::Plugin voodoo, include the HttpHandlerPlugin module, and then use the Configurator#plugin function to set the :handler => plugin() setting.
6.6.1. More Examples
The Mongrel RubyForge project Files section has several other examples of using the GemPlugin system to create distributions of your handlers and commands. You should take a look at mongrel_upload_progress as a full example of implementing a useful set of handlers as GemPlugins.
6.7. Advanced Handlers
Handlers have a “notifications” feature that has been in Mongrel since 0.3.13.3, but which will receive a major change in a version to come after 0.3.13.4. Rather than explain how to use the feature—since it will most likely change—we’ll instead refer you to the mongrel_upload_progress plugin that is available as a stock Mongrel gem.
The mongrel_upload_progress plugin utilizes the notification callbacks to update a visual progress feedback for users uploading files. The example includes Mongrel code and a small Rails example.
6.8. Finding More Plugins
Finding Mongrel plugins isn’t as organized as the Rails plugin system. Mongrel comes with a few that you can see on the RubyForge project Files section. You should refer to the mailing list (http://rubyforge.org/mailman/listinfo/mongrelusers) and watch for any messages announcing new ones.