© Moritz Lenz 2019
Moritz LenzPython Continuous Integration and Deliveryhttps://doi.org/10.1007/978-1-4842-4281-0_7

7. Package Deployment

Moritz Lenz1 
(1)
Fürth, Bayern, Germany
 

In the previous chapters, you have seen how Debian packages are built, inserted into a repository, and how this repository can be configured as a package source on a target machine. With these preparations in mind, interactively installing the actual package becomes easy.

To install the python-matheval sample project, run
$ apt-get update
$ apt-get install python-matheval

on the target machine.

If several machines are required to provide a service, it can be beneficial to coordinate the update, for example, only updating one or two hosts at a time or doing a small integration test on each after moving on to the next. A nice tool for doing that is Ansible ,1 an open source IT automation and configuration management system.

7.1 Ansible: A Primer

Ansible is a very pragmatic and powerful configuration management system that is easy to get started with. If you already know your way around Ansible (or choose to use a different configuration management and deployment system), you can safely skip this section.

Connections and Inventory

Ansible is typically used to connect to one or more remote machines via Secure Shell (SSH) and bring them into a desired state. The connection method is pluggable. Other methods include local, which simply invokes the commands on the local machine instead, and docker, which connects through the Docker daemon to configure a running container. Ansible calls these remote machines hosts.

To tell Ansible where and how to connect, you write an inventory or hosts file. In the inventory file, you can define hosts and groups of hosts, and also set variables that control how to connect to them (Listing 7-1).
# example inventory file
[all:vars]
# variables set here apply to all hosts
ansible_ssh_user=root
[web]
# a group of webservers
www01.example.com
www02.example.com
[app]
# a group of 5 application servers,
# all following the same naming scheme:
app[01:05].example.com
[frontend:children]
# a group that combines the two previous groups
app
web
[database]
# here we override ansible_ssh_user for just one host
db01.example.com ansible_ssh_user=postgres
Listing 7-1

File myinventory: an Ansible Hosts File

See the introduction to inventory files 2 for more information.

To test the connection, you can use the ping module on the command line.
$ ansible -i myinventory web -m ping
www01.example.com | success >> {
    "changed": false,
    "ping": "pong"
}
www02.example.com | success >> {
    "changed": false,
    "ping": "pong"
}

Let’s break the command line down into its components. -i myinventory tells Ansible to use the myinventory file as inventory. web tells Ansible which hosts to work on. It can be a group, as in this example, a single host, or several such things, separated by a colon. For example, www01.example.com:database would select one of the web servers and all the database servers.

Finally, -m ping tells Ansible which module to execute. ping is probably the simplest module. It just sends the response "pong" without conducting any changes on the remote machine, and it is mostly used for debugging the inventory file and credentials.

These commands run in parallel on the different hosts, so the order in which the responses are printed can vary. If a problem occurs when connecting to a host, add the option -vvvv to the command line, to get more output, including any error messages from SSH .

Ansible implicitly gives you the group all, which—you guessed it—contains all the hosts configured in the inventory file.

Modules

Whenever you want to do something on a host through Ansible, you invoke a module to do it. Modules usually take arguments that specify exactly what should happen. On the command line, you can add those arguments with ansible -m module –a 'arguments'. For example:
$ ansible -i myinventory database -m shell -a 'echo "hi there"'
db01.example.com | success | rc=0 >>
hi there

Ansible comes with a wealth of built-in modules and an ecosystem of third-party modules as well. Most modules are idempotent, which means that repeated execution with the same arguments conducts no changes after the first run. For example, instead of instructing Ansible to create a directory, you instruct it to ensure the directory exists. Running such an instruction the first time creates the directory, and running it the second time does nothing, while still reporting success.

Here, I want to present just a few commonly used modules.

The shell Module

The shell module 3 executes a shell command on the host and accepts some options, such as chdir, to change into another working directory, before running the command.
$ ansible -i myinventory database -m shell -e 'pwd chdir=/tmp'
db01.example.com | success | rc=0 >>
/tmp

This is fairly generic, but it is also an option of last resort. If there is a more specific module for the task at hand, you should prefer the more specific module. For example, you could ensure that system users exist using the shell module , but the more specialized user module 4 is much easier to use for that and likely does a better job than an improvised shell script.

The copy Module

With copy ,5 you can copy files verbatim from the local to the remote machine.
$ ansible -i myinventory database -m copy
     -a 'src=README.md dest=/etc/motd mode=644 db01.example.com' | success >> {
    "changed": true,
    "dest": "/etc/motd",
    "gid": 0,
    "group": "root",
    "md5sum": "d41d8cd98f00b204e9800998ecf8427e",
    "mode": "0644",
    "owner": "root",
    "size": 0,
    "state": "file",
    "uid": 0
}

The template Module

template 6 mostly works like copy, but it interprets the source file as a Jinja2 template,7 before transferring it to the remote host. This is commonly used to create configuration files and incorporate information from variables (more on that later).

Templates cannot be used directly from the command line but, rather, in playbooks, so here is an example of a simple playbook.
# file motd.j2
This machine is managed by {{team}}.
# file template-example.yml
---
- hosts: all
  vars:
    team: Slackers
  tasks:
   - template: src=motd.j2 dest=/etc/motd mode=0644

More on playbooks later, but what you can see is that this defines a variable team, sets it to the value Slackers, and the template interpolates this variable.

Running the playbook with
$ ansible-playbook -i myinventory
    --limit database template-example.yml
creates a file /etc/motd on the database server with the contents
This machine is managed by Slackers.

The file Module

The file module 8 manages attributes of file names, such as permissions, but also allows you to create directories and soft and hard links.
$ ansible -i myinventory database -m file
    -a 'path=/etc/apt/sources.list.d
            state=directory mode=0755'
db01.example.com | success >> {
    "changed": false,
    "gid": 0,
    "group": "root",
    "mode": "0755",
    "owner": "root",
    "path": "/etc/apt/sources.list.d",
    "size": 4096,
    "state": "directory",
    "uid": 0
}

The apt Module

On Debian and derived distributions, such as Ubuntu, installing and removing packages is generally done with package managers from the apt family, such as apt-get, aptitude, and, in newer versions, the apt binary directly.

The apt module9 manages this from within Ansible.
$ ansible -i myinventory database -m apt
    -a 'name=screen state=present update_cache=yes'
db01.example.com | success >> {
    "changed": false
}

Here, the screen package was already installed, so the module didn’t change the state of the system.

Separate modules are available for managing apt-keys 10 with which repositories are cryptographically verified and for managing the repositories themselves.11

The yum and zypper Modules

For RPM-based Linux distributions, the yum 12 and zypper modules 13 (at the time of writing, in preview state) are available. They manage package installation via the package managers of the same name.

The package Module

The package module 14 uses whatever package manager it detects. It is, thus, more generic than the apt and yum modules but supports far fewer features. For example, in the case of apt, it does not provide any control over whether to run apt-get update before doing anything else.

Application-Specific Modules

The modules presented so far are fairly close to the system, but there are also modules for achieving common application-specific tasks. Examples include dealing with databases,15 network-related things such as proxies,16 version control systems,17 clustering solutions such as Kubernetes,18 and so on.

Playbooks

Playbooks can contain multiple calls to modules in a defined order and limit their execution to individual hosts or group of hosts. They are written in the YAML file format19, a data serialization file format that is optimized for human readability.

Here is a sample playbook (Listing 7-2) that installs the newest version of the go-agent Debian package, the worker for Go Continuous Delivery (GoCD).20
---
 - hosts: go-agent
   vars:
     go_server: go-server.example.com
   tasks:
   - apt: package=apt-transport-https state=present
   - apt_key:
        url: https://download.gocd.org/GOCD-GPG-KEY.asc
        state: present
        validate_certs: no
   - apt_repository:
        repo: 'deb https://download.gocd.org /'
        state: present
   - apt: update_cache=yes package={{item}} state=present
     with_items:
      - go-agent
      - git
      - build-essential
   - lineinfile:
        dest: /etc/default/go-agent
        regexp: ^GO_SERVER=
        line: GO_SERVER={{ go_server }}
   - copy:
       src: files/guid.txt
       dest: /var/lib/go-agent/config/guid.txt
       user: go
       group: go
   - service: name=go-agent enabled=yes state=started
Listing 7-2

An Ansible Playbook for Installing a GoCD Agent on a Debian-Based System

The top-level element in this file is a one-element list. The single element starts with hosts: go-agent, which limits execution to hosts in the group go-agent. This is the relevant part of the inventory file that goes with it:
[go-agent]
go-worker01.p6c.org
go-worker02.p6c.org

Then it sets the variable go_server to a string, here the hostname where a GoCD server runs.

Finally comes the meat of the playbook: the list of tasks to execute. Each task is a call to a module, some of which have already been discussed. Following is a quick overview.
  • First, apt installs the Debian package apt-transport-https, to make sure that the system can fetch metadata and files from Debian repositories through HTTPS.

  • The next two tasks use the apt_repository 21 and apt_key 22 modules to configure the repository from which the actual go-agent package will be installed.

  • Another call to apt installs the desired package. Also, some more packages are installed with a loop construct.23

  • The lineinfile module 24 searches by regex (regular expression) for a line in a text file and replaces the line it finds with predefined content. Here, we use that to configure the GoCD server that the agent connects to.

  • Finally, the service 25 module starts the agent, if it’s not yet running (state=started), and ensures that it is automatically started on reboot (enabled=yes).

Playbooks are invoked with the ansible-playbook command, for example, ansible-playbook -i inventory go-agent.yml.

There can be more than one list of tasks in a playbook, which is a common use case when they affect different groups of hosts.
---
- hosts: go-agent:go-server
  tasks:
    - apt: package=apt-transport-https state=present
    - apt_key:
        url: https://download.gocd.org/GOCD-GPG-KEY.asc
        state: present
        validate_certs: no
    - apt_repository:
        repo: 'deb https://download.gocd.org /'
        state: present
- hosts: go-agent
  tasks:
    - apt: update_cache=yes package={{item}} state=present
      with_items:
       - go-agent
       - git
       - build-essential
     - ...
- hosts: go-server
  tasks:
    - apt: update_cache=yes package={{item}} state=present
    - apt: update_cache=yes package=go-server state=present
    - ...

Variables

Variables are useful both for controlling flow inside a playbook and for filling out spots in templates to generate configuration files. There are several ways to set variables. One way is to set them directly in playbooks, via vars: ..., as seen previously. Another is to specify them at the command line.
ansible-playbook --extra-vars=variable=value theplaybook.yml

A third, very flexible way is to use the group_vars feature. For each group that a host is in, Ansible looks for a file group_vars/thegroup.yml and for files matching group_vars/thegroup/*.yml. A host can be in several groups at once, which gives you extra flexibility.

For example, you can put each host into two groups, one for the role the host is playing (such as web server, database server, DNS server, etc.), and one for the environment it is in (test, staging, prod). Here is a small example that uses this layout.
# environments
[prod]
www[01:02].example.com
db01.example.com
[test]
db01.test.example.com
www01.test.example.com
# functional roles
[web]
www[01:02].example.com
www01.test.example.com
[db]
db01.example.com
db01.test.example.com
To configure only the test hosts, you can run
ansible-playbook --limit test theplaybook.yml

and put environment-specific variables in group_vars/test.yml and group_vars/prod.yml and web server–specific variables in group_vars/web.yml, etc.

You can use nested data structures in your variables, and if you do, you can configure Ansible to merge those data structures for you, if they are specified in several sources. You can configure this by creating a file called ansible.cfg, with this content:
[defaults]
hash_behavior=merge
That way, you can have a file group_vars/all.yml that sets the default values
# file group_vars/all.yml
myapp:
    domain: example.com
    db:
        host: db.example.com
        username: myappuser
        instance. myapp
and then override individual elements of that nested data structure, for example, in group_vars/test.yml, as follows:
# file group_vars/test.yml
myapp:
    domain: test.example.com
    db:
        hostname: db.test.example.com

The keys that the test group vars file didn’t touch, for example, myapp.db.username, are inherited from the file all.yml.

Roles

Roles are a way to encapsulate parts of a playbook into a reusable component. Let’s consider a real-world example that leads to a simple role definition.

For deploying software, you typically want to deploy the exact version just built, so the relevant part of the playbook is
- apt:
    name: thepackage={{package_version}}
    state: present
    update_cache: yes
    force: yes

But this requires you to supply the package_version variable whenever you run the playbook. This will not be practical when you’re not running a deployment of a freshly built software, but instead you configure a new machine and have to install several software packages, each with its own playbook.

Hence, we generalize the code to deal with a case in which the version number is absent.
- apt:
    name: thepackage={{package_version}}
    state: present
    update_cache: yes
    force: yes
  when: package_version is defined
- apt: name=thepackage state=present update_cache=yes
  when: package_version is undefined

If you include several such playbooks in one and run them on the same host, you’ll likely notice that it spends most of its time running apt-get update for each included playbook.

Updating the apt cache is necessary the first time, because you might have just uploaded a new package on your local Debian mirror prior to the deployment, but subsequent runs are unnecessary. So, you can store the information that a host has already updated for its cache in a fact ,26 which is a kind of per-host variable in Ansible.
- apt: update_cache=yes
  when: apt_cache_updated is undefined
- set_fact:
    apt_cache_updated: true

As you can see, the code base for sensibly installing a package has grown a bit, and it’s time to factor it out into a role .

Roles are collections of YAML files with predefined names. The commands
$ mkdir roles
$ cd roles
$ ansible-galaxy init custom_package_installation
create an empty skeleton for a role named custom_package_installation. The tasks that previously went into all the playbooks now go into the file tasks/main.yml, below the role’s main directory (Listing 7-3).
- apt: update_cache=yes
  when: apt_cache_updated is undefined
- set_fact:
    apt_cache_updated: true
- apt:
    name: {{package}={{package_version}}
    state: present
    update_cache: yes
    force: yes
  when: package_version is defined
- apt: name={{package} state=present update_cache=yes
  when: package_version is undefined
Listing 7-3

File roles/custom_package_installation/tasks/main.yml

To use the role, include it in a playbook like this:
---
- hosts: web
  pre_tasks:
     - # tasks that are executed before the role(s)
  roles:
     role: custom_package_installation
     package: python-matheval
  tasks:
    - # tasks that are executed after the role(s)

pre_tasks and tasks are optional. A playbook consisting only of roles being included is just fine.

Ansible has many more features, such as handlers, that allow you to restart services only once after any changes, dynamic inventories for more flexible server landscapes, Vault for encrypting variables,27 and a rich ecosystem of existing roles for managing common applications and middleware.

For more about Ansible, I highly recommend the excellent book Up and Running, 2nd ed., by Lorin Hochstein (O’Reilly Media, 2017).

7.2 Deploying with Ansible

Armed with knowledge of Ansible from the previous section, deployment becomes a simple task. We start with separate inventory files for the environments (Listing 7-4).
[web]
www01.yourorg.com
www02.yourorg.com
[database]
db01.yourorg.com
[all:vars]
ansible_ssh_user=root
Listing 7-4

Ansible Inventory File production

Perhaps the testing environment requires only a single web server (Listing 7-5).
[web]
www01.testing.yourorg.com
[database]
db01.stagingyourorg.com
[all:vars]
ansible_ssh_user=root
Listing 7-5

Ansible Inventory File testing

Installing the package python-matheval on the web servers in the testing environment is now a one-liner.
$ ansible -i testing web -m apt -a 'name=python-matheval update_cache=yes state=latest'
Once you start deploying with Ansible, it’s likely you’ll want to do other configuration management tasks with it as well, so it makes sense to write a playbook for each package you want to deploy. Here is one (Listing 7-6) using the package installation role from the “Roles” section earlier in this chapter.
---
- hosts: web
  roles:
    role: custom_package_installation
    package: python-matheval
Listing 7-6

File deploy-python-matheval.yml: Deployment Playbook for Package python-matheval

You can then invoke it as
$ ansible-playbook -i testing deploy-python-matheval.yml

7.3 Summary

Ansible can install packages for you, but it can also do much more. It can configure both the operating system and application and even orchestrate processes across several machines.

By writing an inventory file, you tell Ansible which machines it controls. Playbooks specify what to do, using modules to achieve individual tasks, such as creating users or installing software.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset