The pipeline from the previous chapter is already quite usable and vastly preferable to manual builds, distribution, and installation. That said, there is room for improvement. I will discuss how to change it to always deploy the exact version that was built in the same instance of the pipeline, how to run smoke tests after installation, and how to extract a template from the Go continuous delivery (GoCD) configuration, so that it becomes easily reusable.
11.1 Rollbacks and Installing Specific Versions
The deployment pipeline developed in the previous chapters always installs the latest version of a package. Because the logic for constructing version numbers usually produces monotonously increasing version numbers, this is usually the package that was built previously in the same pipeline instance.
However, we really want the pipeline to deploy the exact version that was built inside the same instance of the pipeline. The obvious benefit is that it allows you to rerun older versions of the pipeline, to install older versions, effectively giving you a rollback.
Alternatively, you can build a second pipeline for hotfixes, based on the same Git repository but a different branch. When you want a hotfix, you simply pause the regular pipeline and trigger the hotfix pipeline. In this scenario, if you always installed the newest version, finding a proper version string for the hotfix would be nearly impossible, because it must be higher than the currently installed one but also lower than the next regular build. Oh, and all of that automatically, please.
A less obvious benefit to installing a very specific version is that it detects errors in the package source configuration of the target machines. If the deployment script only installs the newest version that’s available, and through an error the repository isn’t configured on the target machine, the installation process becomes a silent no-op, if the package is already installed in an older version.
Implementation
Ansible Playbook Fragment for Installing Version 1.00 of Package foo
You can invoke this with ansible-playbook --extra-vars=package_version=1.00....
Add this playbook to the deployment-utils Git repository as file ansible/deploy-python-matheval.yml. Finding the version number to install also has a simple, though perhaps not obvious, solution: write the version number to a file; collect this file as an artifact in GoCD; and then, when it’s time to install, fetch the artifact and read the version number from it. At the time of writing, GoCD does not have a more direct way to propagate metadata through pipelines.
GoCD Configuration for Installing the Version from the version File
(The <arg>...</arg> XML tag must be on one line, so that Bash interprets it as a single command. It is shown here on multiple lines merely for readability.)
Bash’s $(...) opens a subprocess, which, again, is a Bash process, and inserts the output from that subprocess into the command line. < ../../version is a short way of reading the file, and this being XML, the less-than sign needs to be escaped.
The production deployment configuration looks pretty much the same, just with --inventory-file=production.
Try It!
To test the version-specific package installation, you must have at least two runs of the pipeline that captured the version artifact. If you don’t have that yet, you can push commits to the source repository, and GoCD picks them up automatically.
You can query the installed version on the target machine with dpkg -l python-matheval. After the last run, the version built in that pipeline instance should be installed.
In the history view of a pipeline, hovering over a complete stage (passed or failed) gives you an icon for rerunning the stage
When the stage has finished running, you can again check the installed version of the package on the target machine, to verify that the older version has indeed been deployed.
11.2 Running Smoke Tests in the Pipeline
When deploying an application, it is important to test that the new version of the application actually works. Typically, this is done through a smoke test—a pretty simple test that nonetheless tests many aspects of the application: that the application process runs, that it binds to the port it is supposed to, and that it can answer requests. Often, this implies that both the configuration and database connection are sane as well.
When to Smoke?
Smoke tests cover a lot of ground at once. A single test might require a working network, correctly configured firewall, web server, application server, database, and so on to work. This is an advantage, because it means that it can detect many classes of errors, but it is also a disadvantage, because it means the diagnostic capabilities are low. When it fails, you don’t know which component is to blame and have to investigate each failure anew.
Smoke tests are also much more expensive than unit tests. They tend to take more time to write, take longer to execute, and are more fragile in the face of configuration or data changes. So, typical advice is to have a low number of smoke tests, maybe one to 20, or maybe about 1% of the unit tests you have.
As an example, if you were to develop a flight search and recommendation engine for the Web, your unit tests would cover different scenarios that the user might encounter and that the engine produces the best possible suggestions. In smoke tests, you would just check that you can enter the starting point, destination, and date of travel, and that you get a list of flight suggestions at all. If there is a membership area on that web site, you would test that you cannot access it without credentials and that you can access it after logging in. So, three smoke tests, give or take.
White Box Smoke Testing
The examples mentioned above are basically black box smoke testing, in that they don’t care about the internals of the application and approach the application just like a user. This is very valuable, because, ultimately, you care about your user’s experience.
Sometimes, there are aspects of the application that aren’t easy to smoke test yet break often enough to warrant automated smoke tests. As an example, the application might cache responses from external services, so simply using a certain functionality is not guaranteed to exercise this particular communication channel.
A practical solution is for the application to offer some kind of self-diagnosis, such as a web page from which the application tests its own configuration for consistency, checks that all the necessary database tables exist, and that external services are reachable. A single smoke test can then call the status page and raise an error whenever the status page either is not reachable or reports an error. This is a white box smoke test.
Status pages for white box smoke tests can be reused in monitoring checks, but it is still a good idea to explicitly check them as part of the deployment process. White box smoke testing should not replace black box smoke testing, but, rather, complement it.
Sample Black Box Smoke Test
The output is the same as before, but the exit status is non-zero, if the output deviates from the expectation.
Adding Smoke Tests to the Pipeline and Rolling Releases
A naive integration of smoke tests in a delivery pipeline is to add a smoke test stage after each deployment stage (that is, one after the test deployment and one after the production deployment). This setup prevents a version of your application from reaching the production environment if it failed smoke tests in the testing environment. Because the smoke test is just a shell command that indicates failure with a non-zero exit status, adding it as a command in your deployment system is trivial.
If you have just one instance of your application running, this is the best you can do. However, if you have a farm of machines, and several instances of the application running behind some kind of load balancer, it is possible to smoke test each instance separately during an upgrade and abort the upgrade if too many instances fail the smoke test.
All big, successful tech companies guard their production systems with such partial upgrades guarded by checks, or even more elaborate versions thereof.
File smoke-tests/python-matheval: A Simple HTTP-Based Smoke Test
File ansible/deploy-python-matheval.yml: A Rolling Deployment Playbook with Integrated Smoke Test
As the number of smoke tests grows over time, it is not practical to cram them all into the Ansible playbook, and doing that also limits reusability. Here, they are instead in a separate file in the deployments utils repository.1 Another option would be to build a package from the smoke tests and install them on the machine that Ansible runs on.
While it would be easy to execute the smoke tests command on the machine on which the service is installed, running it as a local action (that is, on the control host on which the Ansible playbook is started) also tests the network and firewall part and, thus, more realistically mimics the actual usage scenario.
11.3 Configuration Templates
When you have more than one software package to deploy, you build a pipeline for each one. As long as the deployment pipelines are similar enough in structure—mostly using the same packaging format and the same technology for installation—you can reuse the structure, by extracting a template from the first pipeline and instantiating it several times to create separate pipelines of the same structure.
If you look carefully over the pipeline XML configuration developed before, you might notice that it is not very specific to the python-matheval project. Apart from the Debian distribution and the name of the deployment playbook, everything in here can be reused for any software that’s been Debian-packaged.
Parameter Block for the python-matheval Pipeline, to Be Inserted Before the Materials
The next step toward generalization is to move the stages to a template. This can either be done, again, by editing the XML config or in the web interface with Admin ➤ Pipelines and then clicking the Extract Template link next to the pipeline called python-matheval.
GoCD Configuration for Pipeline matheval Using a Template
Everything that’s specific to this one software package is now in the pipeline definition, and the reusable parts are in the template. The sole exception is the deployment-utils repository, which must be added to each pipeline separately, because GoCD has no way to move a material to a template.
Adding a deployment pipeline for another application is now just a matter of specifying the URL, target (that is, name of a group in the Ansible inventory file), and distribution. You will see an example of that in the next chapter. This amounts to fewer than five minutes of work, once you’re used to the tooling.
11.4 Avoiding the Rebuild Stampede
When you have a sizable number of pipelines, you’ll notice an unfortunate pattern. Whenever you push a commit to the deployment-utils repository , it triggers the rebuild of all pipelines. That’s a waste of resources and keeps the build agent or agents occupied, so building of packages based on actual source code changes gets delayed until after all the build jobs have finished.
GoCD Material Definition That Avoids Triggering the Pipeline
The * filter matches all files in the top-level directory, and **/* all files in subdirectories.
When you change the material configuration of the deployment-utils material in all pipelines to have these ignore filters, a new commit to the deployment-utils repository does not trigger any pipelines. GoCD still polls the material and uses the newest version when starting a pipeline. As with all pipelines, the version of the material is the same at all stages.
Using White Lists in GoCD Materials to Selectively Trigger on Changes to Certain Files
Such a white list configuration per pipeline causes commits to the deployment-utils repository to trigger only the pipelines that the changes are relevant for.
11.5 Summary
When you configure your pipelines to deploy exactly the same version that has been built in the same instance of the pipeline, you can use this to install old versions or conduct rollbacks.
Pipeline templates allow you to extract the commonalities between pipelines and maintain those only once. Parameters bring in the variety needed to support diverse software packages.