Time Savings

In medium to large organizations, applications and their infrastructure are typically developed and operated by separate teams. Each deployment must be coordinated between these teams. A change request must be filed, a date must be found that suits both teams, information about the new version must be propagated (such as what new configuration is available or required), the development team must make the binaries available for installation, and so on. All of this can easily consume hours or days of time for each release, both in the development team and in the operations team.

Then the actual deployment process also takes its time, often accompanied by downtimes. And since downtimes are often to be avoided during business hours, the deployments must occur during the night or weekend, making the operations team less eager to perform the task. Precious goodwill is also used up by manual deployments.

Automating deployments can save much time and goodwill. For example, Etsy¹ introduced continuous (and, thus, automated) delivery, reducing the deployment time costs from 6–14 hours by a “deployment army” to a 15-minute effort by a single person.²

Shorter Release Cycles

It is a truism that tasks that take a lot of effort are done much less frequently than those that require virtually no effort. The same is true of risky endeavors: we tend to avoid doing them often.

Companies that do manual releases and deployments often do releases weekly or even less frequently. Some do monthly or even quarterly releases. In more conservative industries, even doing releases every 6 or 12 months is not unheard of.

Infrequent releases invariably lead to drawn-out development processes and slow time-to-market. If software is deployed once every quarter, the time from specification to deployment can easily be dominated by the slow release cycle, at least for small features.

This can mean, for example, that an online business with a bad user experience in the checkout process must wait about three months to improve the user experience, which can cost real money. Automating the deployment makes it easier to release more frequently, alleviating this pain.

Shorter Feedback Cycles

The best way to get feedback for software is to deploy it to the production environment. There, people will actually use it, and you can then listen to what they have to say or even continuously measure their engagement with different parts of the system.

If you are developing tools for internal use in a company, you might get a few people to try them out in a staging environment, but that’s not easy. It takes time from their actual work; the staging environment must be set up with all the necessary data (customer data, inventory, …) even to be usable; and then all the changes there will be lost eventually. In my experience, getting users to test in a non-production environment is hard work and only worth it for major changes.

With manual and, thus, infrequent releases, the feedback cycle is slow, which goes against the whole idea of an “agile” or “lean” development process.

 Lean software development is a development paradigm inspired by Toyota’s lean manufacturing process, which focuses on reducing unnecessary work, delivering software fast, learning, and related principles.

Because human communication is prone to misunderstandings, the first implementation of a feature seldom fulfills the original expectations. Feedback cycles are inevitable. Slow release cycles thus lead to slow development, frustrating both the stakeholders and the developers.

But there are secondary effects too. When improvement cycles take a long time, many users won’t even bother to request small improvements at all. This is a real pity, because a good user interface is made of hundreds of small conveniences and sharp edges that must be rounded. So, in the long run, slow release cycles lead to worse usability and quality.

Reliability of Releases

There is a vicious cycle with manual releases. They tend to be infrequent, which means that many changes go into a single release. This increases the risk of something going wrong. When a big release causes too much trouble, managers and engineers look for ways to improve the reliability of the next release, by adding more verification steps, more process.

But more process means more effort, and more effort leads to even slower cycles, leading to even more changes per release. You can see where this is going.

Automating steps of the release process, or even the whole process, is a way to break this vicious cycle. Computers are much better than humans at following instructions to the letter, and their concentration doesn’t slip at the end of a long night of deploying software.

Once the release process has become more reliable and quicker to execute, it’s easy to push for more frequent releases, each of which introduces fewer changes. The time saved by automation frees resources to further improve the automated release process.

With doing more deployments also comes more experience, which puts you into a good position to improve the process and tools even further.

Smaller Increments Make Triaging Easier

When a deployment introduces a bug, and that deployment introduced only one or two features or bug fixes, it is usually pretty easy to figure out which change caused the bug (triaging). In contrast, when many changes are part of the same deployment, it is much harder to triage new bugs, which means more time wasted, but this also leads to a longer time until defects can be repaired.

More Architectural Freedom

Current trends in the software industry are moving away from huge, monolithic applications toward distributed systems of more and smaller components. This is what the microservice pattern is all about. Smaller applications or services tend to be easier to maintain, and scalability requirements demand that they must each be able to run on different machines, and often on several machines per service.

But if deploying one application or service is already a pain, deploying ten or even a hundred smaller applications promises to be a much bigger pain and makes it downright irresponsible to mix microservices with manual deployment.

Automatic deployments thus open up the space of possible software architectures that you can utilize to solve a business problem.

Advanced Quality Assurance Techniques

Once you have the necessary infrastructure, you can employ amazing strategies for QA. For example, GitHub uses live, parallel execution of new and old implementations³ to avoid regressions, both on the result and performance parameters.

Imagine you develop a travel search engine, and you want to improve the search algorithm. You could deploy both the old and new version of the engine at the same time and run the incoming queries (or a fraction of them) against both and define some metrics by which to evaluate them. For example, fast travel and low costs make a good flight connection. You can use this to find cases in which the new engine performs worse than the old one and use this data to improve it. You can also use this data to demonstrate the superiority of the new search engine, thus justifying the efforts spent developing it.

But such experiments are not practical if each new version must be deployed manually and deploying each version is a major effort. Automatic deployment does not give you these benefits automatically, but it is a prerequisite for employing such advanced QA techniques.

The Pipeline Architecture

A CD system is structured as a pipeline. A new commit or branch in a version control system triggers the instantiation of the pipeline and starts executing the first of a series of stages. When a stage runs successfully, it triggers the next stage. If it fails, the entire pipeline instance stops.

Then manual intervention is necessary, typically by adding a new commit that fixes code or tests or by fixing the environment or the pipeline configuration. A new instance of the pipeline, or a rerun of the failed stage, can then have a chance to succeed.

Deviations from the strict pipeline model are possible. Branches, potentially executed in parallel, for example, allow running different tests in different environments and waiting with the next step until both are completed successfully. Branching into multiple pipelines, and thus parallel execution, is called fan out; joining the pipelines into a single branch is called fan in (Figure 4-1).

The typical stages are building, running the unit tests, deployment to a first test environment, running integration tests there, potentially deployment to and tests in various test environments, and, finally, deployment to production (Figure 4-2).

Sometimes, these stages blur a bit. For example, a typical build of Debian packages also runs the unit tests, which alleviates the need for a separate unit-testing stage. Likewise, if the deployment to an environment runs smoke tests for each host it deploys to, there is no need for a separate smoke test stage (Figure 4-3).

Typically, there is a piece of software that controls the flow of the whole pipeline. It prepares the necessary files for a stage, runs the code associated with the stage, collects its output and artifacts (that is, files that the stage produces and that are worth keeping, such as binaries or test output), determines whether the stage was successful, and then proceeds to the next stage.

From an architectural standpoint, this relieves the stages from having to know what stage comes next and even how to reach the machine on which it runs. It decouples the stages and maintains separation of concerns.

Anti-Pattern: Separate Builds per Environment

If you use a branch model such as GitFlow⁴ for your source code, it is tempting to automatically deploy the develop branch to the testing environment. When the time comes for a release, you merge the development branch into the master branch (possibly through the indirection of separate release branches), and then you automatically build the master branch and deploy the result to the production environment.

It is tempting, because it is a straightforward extension of an existing, proven workflow. Don’t do it.

The big problem with this approach is that you don’t actually test what’s going to be deployed, and on the flip side, you deploy something untested to production. Even if you have a staging environment before deploying to production, you are invalidating all the testing you did, if you don’t actually ship the binary or package that you tested in previous environments.

If you build “testing” and “release” packages from different sources (such as different branches), the resulting binaries will differ. Even if you use the exact same source, building twice is still a bad idea, because many builds aren’t reproducible. Nondeterministic compiler behavior and differences in environments and dependencies all can lead to packages that worked fine in one build and failed in another. It is best to avoid such potential differences and errors by deploying to production exactly the same build that you tested in the testing environments.

Differences in behavior between the environments, where they are desirable, should be implemented by configuration that is not part of the build. It should also be self-evident that the configuration must be under version control and deployed automatically. There are tools that specialize in deploying configuration, such as Puppet, Chef, and Ansible, and later chapters discuss how to integrate them into the deployment process.

Everything Hinges on the Packaging Format

Building a deployable artifact is an early stage in a CD pipeline: build, repository management, installation, and operation all depend on the choice of package format. Python software is usually packaged as a source tarball, in a format determined by the setuptools package, and sometimes as a binary wheel package, specified by the Python Enhancement Proposal (PEP) 427.⁵

Neither source tarballs nor wheels are particularly suitable for deploying a running application. They lack hooks during installation time for creating necessary system resources (such as user accounts), for starting or restarting applications, and other operating system–specific tasks. They also don’t have support for managing non-Python dependencies, such as a database’s client libraries written in C.

Python packages are installed by the pip package manager, which defaults to a system-wide, global installation, which sometimes interacts poorly with Python packages installed by the operating system’s package manager. Workarounds exist, for example, in the form of virtual environment, but managing these requires extra care and effort.

Finally, in the case of separate development and operating responsibilities, the operating team usually is much more familiar with native operating system packages. Nonetheless, source tarballs serve a very useful role as the starting point for creating packages in formats that are more suitable for direct deployment.

In this book, we deploy to Debian GNU/Linux machines, and so we build Debian packages, using a two-step process. First, we create a source tarball using a setup.py file powered by setuptools. Then the tool dh-virtualenv creates a Debian package that contains a virtualenv, into which the software and all of its Python dependencies are installed.

Technology for Managing Debian Repositories

Deploying Debian (and most other) packages works by uploading them into a repository. Target machines are then configured with the URL of this repository. From the perspective of the target machines, this is a pull-based model, which allows them to fetch dependencies that aren’t installed yet. These repositories consist of a certain directory layout, in which files of predefined names and formats contain metadata and link to the actual package files.

These files and directories can be exposed by transport mechanisms, such as local file access (and possibly mounted through a networked file system), HTTP, and FTP. HTTP is good choice, because it is simple to set up, easy to debug, and isn’t usually a performance bottleneck, because it’s a standard system component.

Various software exists to manage Debian repositories, much of which is poorly documented or barely maintained. Some solutions, such as debarchiver, or dak, offer remote upload through SSH but don’t give immediate feedback as to whether the upload was successful. Debarchiver also processes uploaded files in batches, triggered by a cron job, which leads to a delay that makes automation much less fun.

I settled on Aptly ,⁶ which is a command-line toolset for managing repositories. When you add a new package to the repository, Aptly gives immediate feedback in the form of an exit code. It does not provide a convenient way to upload the files onto the server in which the repositories lie, but that is something that the pipeline manager can do.

Finally, Aptly can keep multiple versions of the same package in a repository, which makes it much easier to do a rollback to a previous version.

Tooling for Installing Packages

There are some subtleties to be aware of in automated installations. You must switch off all forms of interactivity, possibly control the verbosity of the output, configure whether downgrades are acceptable, and so on.

Instead of trying to figure out all these details, it is a good idea to reuse an existing tool whose authors have already done the hard work. Configuration-management tools such as Ansible,⁷ Chef,⁸ Puppet,⁹ Salt,¹⁰ and Rex¹¹ have modules for installing packages, so they can be a good choice.

Not all configuration management systems are suitable for automating deployments, however. Puppet is usually used in a pull-based model, in which each Puppet-managed machine periodically contacts a server and asks for its target configuration. That is great for scalability but makes integration into a workflow a major pain. Push-based models, in which the manager contacts the managed machine—for example, through SSH—and then executes a command, are much better suited for deployment tasks (and typically offer a simpler and more pleasant development and debugging experience).

For this book, I’ve chosen Ansible. This is mostly because I like its declarative syntax, its simple model, and that a bit of googling has found good solutions to all practical problems so far.

Controlling the Pipeline

Even if you think of a deployment pipeline in terms of building, testing, distributing, and installing software, much of the work done is actually “glue,” that is, small tasks that make the whole thing run smoothly. These include polling the version control system, preparing the directories for the build jobs, collecting the built packages (or aborting the current pipeline instance on failure), and distributing the work to the machines that are most appropriate for the task.

Of course, there are tools for these tasks as well. General CI and build servers such as Jenkins typically can do the job. But there are also tools specialized in CD pipelines, such as Go continuous delivery (GoCD) ¹² and Concourse.¹³

While Jenkins is a great CI tool, its job-centric worldview makes it less optimal for the pipeline model of CD. Here, we will explore GoCD, which is open source software by ThoughtWorks, Inc. It is written primarily in Java and is available for most operating systems. Conveniently for the Debian-based development environment, it offers pre-built Debian packages.

In the examples in the upcoming chapters, we’ll package a build that also runs the unit tests. In a production setting, you’d likely include a post-build action in the Jenkins pipeline that uses the GoCD API to trigger the CD steps, if all the tests in Jenkins have passed.