Section 5: Production Deployment

5.1 Requirements

5.2 (Too) Many Options

5.3 “Fake” Hardware Layout

5.4 Software Deployment

5.5 Configuration

5.6 Care and Feeding

It’s nearly impossible to tell you how to deploy a real production deployment. Each organization has its own peculiar brand of software, hardware, policies, procedures, people, quirks, and just plain dumb stuff they do on a daily basis. Trying to cover every possible thing you’d run into could (and does) fill up entire bookshelves.

In this section we’ll go through one typical production deployment. We’ll make it platform-agnostic and mention the places where you’ll have to know your stuff about your system. This production deployment will not cover Windows since that’s a whole book in itself and it’s very rare for people to do anything but small- to medium-sized Ruby deployments on Windows.^[3]

What we’ll do is walk through a choice of hardware, software, where the software goes, how best to configure it at first, and how to test each component in stages. After that you’ll have to monitor your deployment to make sure it continues to run and start planning for your next big move should you grow out of this installation. We will leave this part of the configuration to you and other larger books you can find.

5.1. Requirements

You will not do this in a day. If you are expecting to code until 1 minute before your deadline and then simply point and click and have an instant server then you need to take some kind of medication because you are violently hallucinating. You will need at least a week of 8 hours days to make sure your first deployment works and to have the time to do it right.

Yes, that’s right, 40 hours minimum. If you have no Unix system administration experience and no network administration experience then hire someone on a consulting basis and expect them to still take this long. Why would an expert take as long as a beginner? Because they don’t know the particular brand of dumbass at your company. Harsh, but it’s true. If they’re good, they might do it in 20 hours, and if you’re good, then an expert can do it in three. If you’ve done your first deployment in less than that, then you must be forgetting all the time you spent fixing it later.

Because of the wildly differing nature of all the various operating systems possible, we have to gloss over the actual installation of software. We will list any gotchas we or our reviewers found for the various systems, but it’s up to you to use your system’s package installation method.

5.1.1. Required System Access

This is important enough to merit its own subsection. If you do not have sudo or root access to all of these machines, then don’t bother continuing. It is possible to install everything to an unprivileged directory, but that is a major piece of work and isn’t worth the trouble.

5.1.2. Best Practices Rubric

A best practice is most easily defined as “Whatever is working consistently for everyone else.” Your first production deployment is not the time to become clever. What you want to do is adopt some common best practices, but since this short cut doesn’t have the space to tell you what those are, we’ll give you a little rubric to see if you’re ready.

1. Does your application have full unit testing?
2. Are you doing automated daily deployments to a staging server?
3. Does your planned production deployment have agreement from everyone that it should goout?
4. Do you have either a QA team that can test it or automated “interaction tests” with a tool such as Watir or Selenium?
5. Do you already have the hardware installed, configured with your chosen OS, and networked properly?
6. Have you tested that the network is configured correctly and fast enough for your deployment?
7. Do you have the proper amount of RAM and disk for your deployment?
8. Is Ruby installed on all machines such that it can build Mongrel properly?
9. Is there a dedicated person whose sole responsibility is building and maintaining the production deployment?
10. Does this person have a dedicated developer to help or is he already a developer familiar with the code?
11. Do you have a shared location where you can document the deployment, such as a Wiki or CMS?
12. Do you know how to use httperf or ab and know what the statistics mean?
13. Are you using Rails migrations for all of your database changes and have scripted anything that migrations can’t do?

For each question you answer with “NO,” add 10 hours to your time estimate for completion. This may seem unrealistic, since saying “NO” to everything means it’ll take 190 hours (about one month), but this estimate is actually low according to most first deployment experiences. What we’ve found is that without all these things in place, you end up doing a poor deployment and spend the next month fixing it up.

If you’re just starting out and can’t answer these questions properly, then consider these as instructions for what you need to do for a good development and staging setup. Go through the pains of getting to “YES” for each of these questions in your staging setup and then do your production deployment.

5.1.3. Worst Practices

---

Zed Sez

Programmers recently had a major shift in their quality focus with the introduction of best practices from the Agile methodologies and the Capability Maturity Model. These days if you aren’t doing continuous integration, writing solid unit tests, performing personal process improvement, frequently communicating with your team, and automating everything, then you are easily labeled an amateur developer.

Nothing like this has happened for system administrators. I’m not quite sure how this half of the IT industry gets away with it, but my experience has been that system administrators are far behind in their quality practices compared to developers. They frequently do things by hand, never measure the quality of their systems, are constantly making unilateral changes that impact the company, and frequently make policy decisions in the name of “security.” Usually these “security” policies only make their unautomated job easier rather than actually improve security.^[4]

Sure, developers screw up too, but today developers who haven’t adopted many quality practices are ridiculed and laughed at by their peers. Nobody questions why a developer can build, test, deploy, and validate an entire product with one script, yet a system administrator has to spend two hours typing to do simple things like change passwords.

If you’re a system administrator, the best advice I can give is “Automate or die.” I firmly believe that 99% of what system administrators do can be automated since I’ve done just that everywhere I’ve worked. Watch out because there’s money in wiping you out with automation. It just takes one good book written by the Kent Beck of system administration and you’re out of a job.

If you’re a system administrator and you’re offended by this, then answer this question: “Could you not run ssh for a week?” No, I don’t mean write a script named “flabber” that just runs ssh. I mean is your administration automated to the point that you don’t need to ssh manually into your machines for weeks on end? If you log in hourly or even daily, then you are part of the problem. You have no right to be offended.

---

The following list of “worst practices” is for both developers and system administrators, but it’s more for system administrators since they’ll be doing the deployments. I’ve done all of these at some point, but I don’t do them anymore.

1. Creating and maintaining unversioned files.
2. Doing config changes on production, manually rather than through the automated deploy process
3. Not telling everyone you’re making changes.
4. Deploying without telling anyone.
5. No strict development, staging, production-automated maintenance process.
6. No production rollout-approval process.
7. Not indicating which changes are going into production.
8. Not following best practices (you are not smarter than I am).
9. Not following the same process for dev, staging, and prod every time.
10. Letting anyone deploy to production.
11. Not letting anyone deploy to development.
12. Preventing developers from controlling staging and development.
13. Not collaborating on the staging deployment so they know what’s going on.
14. Not documenting their systems and deployment methods.
15. Not tracking quality of service to identify production problems and proactively prevent them.
16. Not automating everything done to any production system.
17. Using substandard tools because they are either free or commercial (people should use the best tool, not the one with the most CYA^[5] factor).

If you do any of this, then stop now. You are not helping, and it’s just going to add even more problems in the future. I would say if you do any of the above practices, add another 10 hours for each practice.

5.2. (Too) Many Options

A common complaint about deploying Mongrel is the large number of available deployment options. You can use Pound, Pen, Balance, Apache, IIS, Nginx, LightTPD, Litespeed, and probably servers we’ve never even heard of before. You can put Mongrel behind firewalls and hardware load balancers. It’s very daunting for the beginner.

What we’re going to cover is a good, solid best-practice deployment involving Apache, Mongrel, and MySQL on separate servers. You could also combine these three pieces onto one machine, but to make things interesting, let’s configure three machines.

5.3. “Fake” Hardware Layout

What we want is three machines that will serve one purpose each. One machine (web1) will run Apache and be the only Internet facing piece of hardware (and should go in a DMZ as shown in Figure 5.1). The next machine (app1) will run Mongrel and service Rails requests to Apache. The last machine (db1) will run MySQL to maintain the database.

Figure 5.1. Sample Network Configuration

In reality you’ll probably have to test this configuration and mix it up a bit. You might need a couple of Mongrel machines or you might be able to use just one machine for everything. For education purposes, though, we’ll just stick with this as if it were real.

Additionally, you’ll want to avoid configuring a firewall setup like in Figure 5.1 until you have a fully functioning and tested deployment and you know its performance. Firewalls are great for security, but when you’re first setting up a deployment, they can complicate things unnecessarily. We’ve seen many deployments go wrong because of poorly configured firewalls, and knowing that the configuration worked before adding the firewalls is a big help. Reduce lurking variables (http://en.wikipedia.org/wiki/Lurking_variable) so you can focus on problems in isolation.

5.4. Software Deployment

Every operating system has different ways to deploy software, even different versions of the same operating system. Some have a choice between source build and package management. Commercial systems have restrictions on what is an approved software package and version and installing anything else means you don’t get support.

Because of this, it’s not really possible for us to explain how to install the software on your system. We’ll give you the main components that belong on each machine, and leave it to you to make sure they get installed correctly.^[6] We’ll also include some common problems people run into when they install each component.

5.4.1. web1 (Apache)

Apache 2.2.x You need this more recent version of Apache so you can get the new faster and more capable mod_proxy_balancer. There was a defect in the 2.2.2 version of Apache that caused the proxy code to fail in some cases, so make sure your version is either higher numbered or doesn’t have this problem. Our example in Section 4 uses 2.2.3.

Apache 2.0.x (alternative) Some operations departments do not allow the latest bleeding edge. The problem is that Apache versions before 2.2.x had a slow mod_proxy, which didn’t support multiple backends. The alternative is to install Apache 2.0.x on the web1 machine, and then Balance on the app1 machine.

Once Apache is installed you’ll want to make sure it’s part of your system’s boot process, that you can start/stop it, and that it’s accessible on port 80. If you need SSL, then take the time to learn how to configure this and get your certificate as well. We won’t be covering SSL configuration in this short cut.

5.4.2. app1 (Mongrel)

Ruby Make sure you include all tools necessary to build C extensions. For example, Debian requires build-essentials and several Ruby-related libraries.

RubyGems Some systems are now including this, but most people seem to install RubyGems from source download. You can get it from RubyForge http://www.rubyforge.org/projects/rubygems/.

Rails This is pretty easy with RubyGems, but systems are now also installing it through the package management.

Mongrel You of course need Mongrel, and there are also some systems that package it, so you may be in luck.

libmysqlclient You’ll need to install whatever your OS calls the MySQL client libraries. Many times it’s easier to just install MySQL (disk is cheap).

mysql gem You’ll need the Ruby library from RubyGems to talk to MySQL, which also means you’ll need the client libraries on the app1 box. This is best installed using gem install mysql rather than your package system.

balance (alternative) If you are forced to use Apache 2.0.x, then you’ll need to put balance on this machine. Balance simply takes TCP/IP connections to one port and proxies them to a set of backend ports. It’s fast and relatively easy to set up.

This is in addition to any software you need to get your Rails application working. My recommendation is that you install only the bare minimum to get a simple test Rails application working. Only after everything is configured and running well should you install your main application.

We typically install Ruby using the OS package system and then RubyGems if it is available. After that we install everything else using gem install. Depending on your purity you may try to install everything through your system’s packages or mix it up like we do.

5.4.3. db1 (MySQL)

Obviously you’ll want to install MySQL on the db1 machine and configure it properly. Hopefully you’ve been using MySQL this whole time and doing an install is second nature to you. It might be difficult to configure MySQL for production without some prior knowledge, but ask around for a consultant to help you out. There are also a few good books you can use as a reference.

We have to confess, though, that we usually install Webmin or phpMyAdmin to manage the MySQL server. System administrators who are die-hard script junkies hate these tools, but they encapsulate many system management best practices and can help someone who has limited system administration experience. Consider installing Webmin if you need to administer a machine and don’t have much time to monkey around with configuration files. An additional advantage of Webmin is that you can cluster your management and manage a fairly large group of machines from one primary machine.

5.4.4. All Boxes

Ruby If you are looking to do Capistrano management of all three boxes, then you’ll need Ruby on all three. Capistrano is super good stuff, so you’ll want to make this leap very soon.

5.5. Configuration

5.5.1. Mongrel^[7]

We’re going to use mongrel_cluster to manage your small set of Mongrel applications. We’ll need the mongrel_cluster gem, and we’ll first just set up the testapp so you can see how it’s done. This will lay the groundwork for your real production application.

5.5.2. A Simple Test Rails Application

There’s a bit of a chicken and eggs problem with this kind of setup. In order to test that your web1 machine works with your app1 machine, you need to have a working application on app1. But you can’t really get your application working without web1 and db1 working. The best thing to do is to create a small “test” application and serve it manually until you’re ready for the big time:

You should then be able to visit http://localhost:3000/ and see the Rails test page. That means your Mongrel setup is working, but you’ll want something that also shows Rails is working. Stop Mongrel with CTRL-C and do:

$ script/generate controller test

Edit app/controllers/test_controller.rb and add this to the TestController class:

Then start up Mongrel again and go to http://localhost:3000/test to see your little tester Rails application.

5.5.3. mongrel_cluster

If you haven’t yet, stop your Mongrel instance with CTRL-C and install the mongrel_cluster gem:

$ gem install mongrel_cluster

Next we’ll want to configure our testapp instance so that it is running three instances of mongrel starting at port 8000:

This creates a configuration file mongrel_cluster.yml and puts it in /var/www/apps/testapp/config. Now you can start this cluster using:

$ cd /var/www/apps/testapp
$ mongrel_rails cluster::start

And of course the command cluster::stop will stop it. You should now (with the cluster still running) try to hit each port (8000, 8001, 8002) with your browser and make sure the test pages are present.

We’re going to leave this running in this for now while we set up the web1 Apache server. Then we’ll come back and finalize the install by putting your app on, configuring it, and putting it into the boot process for app1.

5.5.4. Apache

Refer to Section 4 to learn how to get your Apache 2.2.x install up and running. If you’re using virtual hosts then make sure they are configured correctly and serving the files for your domain name.

Once you have everything running you’ll need to change your configuration to point your virtual host at the app1 servers:

Change app1 to be either your real hostname or the IP address. Restart your Apache instance according to your system’s process restart method and hit http://web1/ to see if your application comes up. If not, double-check the following:

1. Can you access ports 8000, 8001, 8002 from web1 to app1? Use curl http://app1:8000/ to test it quickly.
2. Are you using fancy rewrite rules? Disable or remove your rewrite rules for now. They aren’t doing you any good at this stage since the app1 server is in a different location and they only add complexity.
3. Can you even access http://web1/ normally?
4. Is it plugged in? No, seriously, are all the machines turned on and networked properly? Hopefully you aren’t trying to do this in the midst of a firewall configuration, so try to ssh to each machine and make sure they are accessible.
5. Did you check the Apache server logs for errors?
6. Did you check the Mongrel server mongrel.log and production.log files? There are usually clues in there.
7. If you get really desperate, shut down your cluster and refer to Dash-Bee Logging in Section 7, then shut down your testapp cluster. With the cluster down you can start up one Mongrel instance on port 8000 with: mongrel_rails start -e production -p 8000 -B. Run this and then try accessing your Apache server again. See if you get any information in log/mongrel_debug/rails.log about the requests.

5.5.5. MySQL

You now have your web1 server balancing against your app1 Mongrel cluster and you’re ready to get your database configured on db1. First thing, though, is you should have been using Rails migrations this whole time. It was mentioned in Section 5.1 as something you should do, but it’s pretty much a requirement at this point. Later you’ll have to deploy your schema to MySQL and then manage versioning of the schema, so if you want to be a hero and do them manually, make sure your SQL scripts are set up properly.

If you don’t have migrations, you’re on your own (and probably very much in trouble). You’ll have to go through the MySQL configuration and just make it up as you go.

What you need to do is first make sure MySQL is running and configured properly and that you can use the mysql tool to connect to localhost on db1. Once you can do that, follow these steps:

1. Set up the MySQL user you plan to use for your production deployment on db1.
2. From app1 use the mysql tool to try connecting to db1 and run a few queries. You may have to install this tool on app1 if you went the purist’s route and only installed the mysql client libraries.
3. Once these two machines are talking mysql protocol, you’re ready to install your production application and get it working.

Until you can connect manually from app1 to db1, you shouldn’t continue. Permissioning and user management in MySQL is rather weird and painful (well, with any DBMS), so we usually break down and install phpMyAdmin or webmin to manage it. Yeah, we’re evil.

5.5.6. Last Step: Your Application in Production

These instructions are intended to get you up and running quick with a basic best practice deployment, but also teach you how everything is installed and managed. In reality everyone who’s half smart about anything uses Capistrano to do their deployments. Capistrano is excellent software that uses Rake and ssh to automate application deployments. It can manage fairly large installations, restart servers, roll back failed deployments, and provide status as well.

The problem with Capistrano is that until you’ve done one deployment manually it’s difficult to use Capistrano to automate your installation. We usually do the first deployment by hand in a new location, and then we step back and plan the automation with Capistrano. As you gain experience doing this you’ll be able to start off right away with Capistrano, but for now we’re going to do this first deployment by hand.

It is a sin of the highest degree to continue doing deployments manually once you’ve done your first one by hand. Capistrano is a fantastic tool that is easy to understand if you know even a small amount about build tools (Rake, Make, SCons, nmake). Taking the time after this deployment to automate your work will save you much pain and anger in the future.

5.5.6.1. Get Your Application On

You’ve got two approaches to follow to get your code onto the app1 server:

1. From your development system just copy it over to app1:/var/www/apps/myapp using scp or rsync. You should really only do this if your app1 machine cannot access your version control repository because of firewall issues.
2. Log onto app1 and checkout your application from version control directly into the /var/www/apps/myapp directory. This is the preferred way since this is how Capistrano does things. In fact, only do option #1 in desperation since you’ll have to change things later to accommodate Capistrano.

Let us assume that you’ve got your code in Subversion and you’re going to set up your application just like your testapp:

Your application should start up and you should see tons of errors. What you must do now is configure this deployment so that it uses your db1 database, runs your rake db:migrate task, and completes any configurations your application specifically needs. It might be a good idea during this stage to not use mongrel_cluster but to just run the application manually with mongrel_rails start -p 8000 and test against http://app1:8000/ until the application works.

Halt! Before you start to change the files you just checked out, take a step back. You have a revision control tool that helps you keep track of your source changes, but if you edit files outside of this process then you’ll run into conflicts later. Since you’ve checked the source out, you have three choices that make sense:

1. Consider your checkout a “development action” and plan on checking your changes in when you are done. This means you have to coordinate with the other developers and make sure your changes don’t break their build.
2. Consider this a “deployment action” and that it should remain untouched. In this case you would do development on another machine, check the changes in, and then svn update on the app1 machine.
3. Don’t do a svn co and instead do an svn export to get a copy of the source that can’t be changed. This is the safest approach but also kind of a pain.

Our general process is to do #1 for the first deployment since it’s quicker and more direct, then when we automate with Capistrano we use #3 to keep things sane and safe.^[8]

5.5.6.2. Systemize Your Deployment

The mongrel_cluster gem comes with an /etc/init.d startup script that works on most linux systems. There’s a small process you can do so that your newly minted and functioning application will be started whenever the machine reboots.

Then you’ll need to configure your system to run /etc/init.d/mongrel_cluster on start/stop and you’re set. You should try rebooting your machine to make sure the application actually does start properly.

5.5.6.3. The Final Get-Together

The only remaining task is to actually make sure all the pieces work. It’s a good idea to test each component in order from back to front, and then test them as one whole. If you were smart you’d also automate this test process and include it in your post-production deployment process.

5.5.6.4. Cheap Simple Caching

Normally you’ll need to set up Apache mod_rewrite rules of varying complexity levels to take advantage of Rails’ page caching. Since you have a front-facing web1 server you’ll have to do some research to find a way to share disk with app1. This is fairly complex but there is a quick and dirty way around it for many people’s applications: mod_cache.

This configuration is only a sample taken from the official mod_cache documentation, but it demonstrates the trick. This will cache the majority of your content and will work as long as your content doesn’t require immediate updating. There are ways to also exclude some locations, to cache to disk, and to control the cache organization. Using the memory cache often works well, as you can just bounce the server after a deployment to reset it, but the details of your application will determine if it is good fit for you.

Another trick you should investigate is using an asset server in Ruby on Rails. You make this setting in config/environments/production.rb:

config.action_controller.asset_host = "http://assets.web1"

This tells Rails to write URIs for assets (Javascript, images, stylesheets, etc.) so that they point to a server named “assets.web1.” If you then set up a virtual host for web1 to map this, you will be able to put a majority of your assets on that server and have them served directly rather than from app1.

5.6. Care and Feeding

5.6.1. Monitoring

You’ll want to install some form of monitoring for all of your systems. Section 5.6.2 lists several good open-source monitoring tools for both host-based and network-based monitoring. You should also take a look at monit as a means of watching your systems internally (meaning not from an external server the way nagios would do it). Others prefer to use runit to manage their processes reliably, and it should work well with monit.

5.6.2. Security

You can install a few small pieces of software right away and get a massive security improvement.

samhain A fantastic little bit of software that monitors your system for host-based intrusion. It can detect file changes, root kits, logins, logouts, privilege escalation, and use a secure central server or e-mail for reporting. It’s also great for finding rogue admins and developers who change things without telling anyone.

psad This or portsentry will watch your system for portscans (people trying to find out what your systems are running) and then block them actively. This is a very effective way of stopping most attackers since the first thing they do is scan your machine.

mod_security This is a module for Apache that lets you do very capable security controls at the HTTP level. You can sometimes use this to protect against new exploits found for Rails without having to upgrade right away.

snort Another fantastic tool that watches your network for potential attacks. You can put it on a separate little machine (running OpenBSD if you’re ultra hardcore) and it’ll detect network level intrusions.

syslog-ng A great way to centralize your system logging. A centralized log infrastructure makes it easier to recover from intrusions, find out how they were done, and makes it easier to monitor your systems.