In the following chapters, we will explore the tool Go continuous delivery (GoCD) and how to package, distribute, and deploy software with it. If you want to follow along and experiment with the things described in these chapters, you will require some machines on which you can do that.
If you don’t have the luxury of a public or private cloud in which you can run virtual machines (VMs) and try out the examples, you can use the tools introduced in this chapter to create a playground of VMs on your laptop or workstation. Even if you do have access to a cloud solution, you might want to use some of the scripts presented here, to set up and configure these machines.
8.1 Requirements and Resource Usage
Build Debian packages.
Upload them to a local Debian repository.
Install the packages on one or more servers.
Run some lean and simple web services.
Run deployment and configuration scripts with Ansible.
Control everything via a GoCD server and agent.
Except for the last task, all of these tasks require very few resources. The GoCD server requires the most resources, with 1GB of RAM minimally and 2GB recommended. The Go server is also the one system that keeps persistent state (such as configuration and pipeline history) that you typically don’t want to lose.
So, the easiest approach, and the one I’m taking here, is to install the Go server on the host machine, which is the laptop or workstation that I typically work with.
Then there is one VM for running the Go agent, on which the Debian packages will be built. Two more VMs serve as the target machines on which the freshly built packages will be installed and tested. One of them serves as a testing environment, the second as a production environment.
For those three VMs, the defaults of a half GB of RAM that the tooling provides is quite sufficient. If you use this playground and don’t have enough RAM on the host machine, you can try to halve the RAM usage of those VMs. For the target machines, even 200MB might be enough to get started.
8.2 Introducing Vagrant
Vagrant is an abstraction layer over classical virtualization solutions, such as KVM and VirtualBox. It offers base images (called boxes) for your VM, manages the VMs for you, and offers a unified API for the initial configuration. It also creates a virtual private network that allows the host machines to talk to the VMs and vice versa.
To install Vagrant, you can download the installer from www.vagrantup.com/downloads.html , or if you use an operating system with a package manager, such as Debian or RedHat, you can install it through the package manager. On Debian and Ubuntu, you would install it with apt-get install vagrant (though avoid the 2.0 series and install 2.1 or newer versions from Vagrant’s web site, if only 2.0 is available through the package manger).
This installs a Vagrant plug-in that automatically installs guest tools inside Vagrant boxes, which improves configurability and reliability.
To use Vagrant, you write a small Ruby script called Vagrantfile , which instantiates one or more boxes as VMs. You can configure port forwarding, private or bridged networks, and share directories between the host and guest VMs.
The vagrant command-line tool allows you to create and provision the VMs with the vagrant up command, connect to a VM with vagrant ssh, obtain a status summary with vagrant status, and stop and delete the VMs again with vagrant destroy. Calling vagrant without any arguments gives you a summary of the options available.
You might wonder why we use Vagrant VMs instead of Docker containers. The reason is that Docker is optimized to run a single process or process group. But for our use case, we have to run at least three processes: the GoCD agent and the application that we actually want to run there; Aptly, to manage a Debian repository; and an HTTP server, to allow remote access to the repository.
Network and Vagrant Setup
We’ll use Vagrant with a virtual private IP network with addresses from 172.28.128.1 to 172.28.128.254. When you assign one or more addresses of this range to a VM, Vagrant automatically assigns the host machine the address 172.28.128.1.
Do not do this for production machines. This is only safe on a virtual network on a single machine, with which you can be sure that no attacker is present, unless they have already compromised your machine.
Creating and destroying VMs is common in Vagrant land, and each time you create them anew, they will have new host keys. Without such a configuration, you’d spend a lot of time updating SSH key fingerprints.
The Vagrantfile and Ansible playbook introduced here can be found in the deployment-utils repository on GitHub in the playground folder. To follow along, you can use it like this:
$ git clone https://github.com/python-ci-cd/deployment-utils.git
$ cd deployment-utils/playground
$ vagrant up
$ ansible-playbook setup.yml
Listing 8-1 shows the Vagrantfile that creates the boxes for the virtual playground.
Vagrantfile for the Playground
This Vagrantfile assumes that you have an SSH key pair, and the public key is inside the .ssh/id_rsa.pub path below your home directory, which is the default location for RSA SSH keys on Linux. It uses Vagrant’s shell provisioner to add the public key to the authorized_keys file of the root user inside the VMs, so that you can log in via SSH on the guest machines. (Vagrant offers a vagrant ssh command for connecting without this extra step, but I find it easier to use the system ssh command directly, mostly because it is not tied to the presence of the Vagrantfile inside the current working directory.)
to spin up and provision the three VMs. It takes a few minutes when you do it the first time, because Vagrant has to download the base box first.
Now you can log in to the VMs with ssh [email protected] and with testing.local and production.local as host names.
8.3 Configuring the Machines
ansible.cfg: A Configuration File for the Playground
Disabling host key checking should only be done in trusted virtual networks for development systems and never in a production setting.
hosts Inventory File for the Playground
File setup.yml: An Ansible Playbook for Configuring the Three VMs
- Installs and configures the GoCD agent
It copies a file with a fixed UID to the configuration directory of the Go agent, so that when you tear down the machine and create it anew, the Go server will identify it as the same agent as before.
- Gives the go user on the go-agent machine SSH access on the target hosts by
First making sure the Go user has an SSH key
Copying the public SSH key to the host machine
Later distributing it to the target machines using the authorized_key module
- Creates a GPG key pair for the go user
Because GPG key creation uses lots of entropy for random numbers, and VMs typically don’t have that much entropy, it first installs rng-tools and uses that to convince the system to use lower-quality randomness. Again, this is something you should never do in a production setting.
Copies the public key of said GPG key pair to the host machine and distributes it to the target machines using the apt_key module
- Creates some Aptly-based Debian repositories on the go-agent machine by
Copying the add-package script from the same repository to the go-agent machine
Running it with a dummy package to actually create the repositories
Installing and configuring lighttpd to serve these packages over HTTP
Configuring the target machines to use these repositories as a package source
Checks that the Go user on the go-agent machine can indeed reach the other VMs via SSH
After running the playbook with ansible-playbook setup.yml, you have a GoCD agent waiting to connect to a server. Installing a GoCD server is covered in the next chapter. After installing the GoCD server, you have to activate the agent in the web configuration and assign the appropriate resources (debian-stretch, build, and aptly, if you follow the examples from this book).
8.4 Summary
Vagrant helps you to set up a virtual playground for CD by managing VMs and a private network. We have seen an Ansible playbook that configures these machines to provide all the infrastructure you need to run a GoCD server on the host machine.