© Moritz Lenz 2019
Moritz LenzPython Continuous Integration and Deliveryhttps://doi.org/10.1007/978-1-4842-4281-0_10

10. Distributing and Deploying Packages in the Pipeline

Moritz Lenz1 
(1)
Fürth, Bayern, Germany
 

The previous chapter has left us with the beginning of a GoCD pipeline. It automatically builds a Debian package each time new commits are pushed to Git and generates a unique version number for each build. Finally, it captures as artifacts the built package and a file called version containing the version number. The next tasks are to upload it into a Debian repository and deploy it on the target machines.

10.1 Uploading in the Pipeline

Chapter 6, on distributing packages, already introduced a small program for creating and filling Debian repositories managed with Aptly. If you add it to the deployment-utils Git repository from that chapter, you can automatically upload the newly built packages with this additional GoCD configuration (Listing 10-1), to be inserted after the build stage.
<stage name="upload-testing">
  <jobs>
    <job name="upload-testing">
      <tasks>
        <fetchartifact pipeline="" stage="build"
            job="build-deb" srcdir="debian-packages"
            artifactOrigin="gocd">
          <runif status="passed" />
        </fetchartifact>
        <exec command="/bin/bash">
          <arg>-c</arg>
  <arg>deployment-utils/add-package testing stretch *.deb</arg>
        </exec>
      </tasks>
      <resources>
        <resource>aptly</resource>
      </resources>
    </job>
  </jobs>
</stage>
Listing 10-1

GoCD Configuration for Uploading a Freshly Built Package to the testing Repository

The fetchartifact task fetches, you guessed it, an artifact that is stored in the GoCD server’s artifact repository. Here, it fetches the directory python-matheval, into which the previous stage uploaded the Debian package. The empty string for the pipeline name instructs GoCD to use the current pipeline.

In the invocation of the add-package script, testing refers to the name of the environment (which you can choose freely, as long as you are consistent), not the testing distribution of the Debian project.

Finally, the aptly resource selects a GoCD agent with the same resource to run the job on (see Figure 10-1). If you anticipate that your setup will grow a bit, you should have a separate machine for serving these repositories. Install a GoCD agent on it and assign it this resource. You can even have separate machines for the testing and production repositories and give them more specific resources (such as aptly-testing and aptly-production).
../images/456760_1_En_10_Chapter/456760_1_En_10_Fig1_HTML.jpg
Figure 10-1

The machine on which the Aptly repository resides has a GoCD agent that retrieves the Debian packages as artifacts from the GoCD server. Target machines configure the repository as a package source.

User Accounts and Security

In the previous sample configuration, the add-package script runs as the go system user, whose home directory on Linux-based systems is /var/go by default. This will create repositories in a directory such as /var/go/aptly/testing/stretch/.

In Chapter 6, the assumption was that Aptly runs under its own system user account. You still have to give the go user permissions to add packages to the repository, but you can prevent the go user from modifying existing repositories and, more important, from getting access from the GPG key with which the packages are signed.

If you keep the repository under a separate user, you need a way to cross the user account barrier, and the traditional way to do that for command-line applications is to allow the go user to call add-package through the sudo command. But to get an actual security benefit, you have to copy the add-package command to a location where the go user has no write permissions. Otherwise, an attacker with access to the go user account could just modify this command to do whatever he/she sees fit.

Assuming you intend to copy it to /usr/local/bin, you can add this line:
/etc/sudoers
to the file (Listing 10-2).
go ALL=(aptly) NOPASSWD: /usr/local/bin/add-package
Listing 10-2

/etc/sudoers Line That Allows the go User to Execute add-package As User aptly

Then, instead of calling add-package <environment> <distribution> <deb package>, you change it to
$ sudo -u aptly --set-home /usr/local/bin/add-package
    <environment> <distribution> <deb package>

The --set-home flags tells sudo to set the HOME environment variable to the home directory of the target user, here aptly.

If you choose not to go the sudo route, you have to adapt the web server configuration to serve files from /var/go/aptly/ instead of /home/aptly/aptly.

10.2 Deploying in the Pipeline

In Chapter 7, we saw how to upgrade (or install, if it’s not yet installed) a package through Ansible (see Figure 10-2), as follows:
$ ansible -i testing web -m apt
    -a 'name=python-matheval state=latest update_cache=yes'
where testing is the inventory file of the same name as the environment, web is the group of hosts to deploy to, and python-matheval is the name of the package.
../images/456760_1_En_10_Chapter/456760_1_En_10_Fig2_HTML.jpg
Figure 10-2

The GoCD agent runs Ansible to connect to the target machines via SSH, to install the desired package

You can do this in GoCD as a separate stage, after the upload-testing stage (Listing 10-3).
<stage name="deploy-testing">
  <jobs>
    <job name="deploy-testing">
      <tasks>
        <exec command="ansible" workingdir="deployment-utils/ansible/">
          <arg>--inventory-file=testing</arg>
          <arg>web</arg>
          <arg>-m</arg>
          <arg>apt</arg>
          <arg>-a</arg>
          <arg>name=python-matheval state=latest update_cache=yes</arg>
          <runif status="passed" />
        </exec>
      </tasks>
    </job>
  </jobs>
</stage>
Listing 10-3

GoCD Configuration for Automatically Installing a Package

This assumes that you add the inventory files in the ansible directory of the deployment-utils Git repository, and that the Debian repository is already configured on the target machine, as discussed in Chapter 7.

10.3 Results

To run the new stage, either trigger a complete run of the pipeline by hitting the “play” triangle in the pipeline overview on the web front end or do a manual trigger of that one stage in the pipe history view. You can log in on the target machine, to check if the package was successfully installed.
$ dpkg -l python-matheval
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name          Version     Architecture Description
+++-==============-============-============-==============
ii python-matheval  0.1-0.7.1   all         Web service
and verify that the service is running
$ systemctl status python-matheval
  python-matheval.service - Package installation informati
   Loaded: loaded (/lib/systemd/system/python-matheval.ser
   Active: active (running) since Sun 2016-03-27 13:15:41
  Process: 4439 ExecStop=/usr/bin/hypnotoad -s /usr/lib/py
 Main PID: 4442 (/usr/lib/packag)
   CGroup: /system.slice/python-matheval.service
           ├─4442 /usr/lib/python-matheval/python-matheval
           ├─4445 /usr/lib/python-matheval/python-matheval
           ├─4446 /usr/lib/python-matheval/python-matheval
           ├─4447 /usr/lib/python-matheval/python-matheval
           └─4448 /usr/lib/python-matheval/python-matheval
You can also check from the host machine that the service responds on port 8080, as it’s supposed to.
$ curl --data '["+", 5]' -XPOST http://172.28.128.3:8800
5

10.4 Going All the Way to Production

Uploading and deploying to production works the same as with the testing environment. So, all that’s required is to duplicate the configuration of the last two pipelines, replace every occurrence of testing with production, and add a manual approval button, so that production deployment remains a conscious decision (Listing 10-4).
<stage name="upload-production">
  <approval type="manual" />
  <jobs>
    <job name="upload-production">
      <tasks>
        <fetchartifact pipeline="" stage="build" job="build-deb" srcdir="debian-packages" artifactOrigin="gocd">
          <runif status="passed" />
        </fetchartifact>
        <exec command="/bin/bash">
          <arg>-c</arg>
          <arg>deployment-utils/add-package production stretch *.deb</arg>
        </exec>
      </tasks>
      <resources>
        <resource>aptly</resource>
      </resources>
    </job>
  </jobs>
</stage>
<stage name="deploy-production">
  <jobs>
    <job name="deploy-production">
      <tasks>
        <exec command="ansible" workingdir="deployment-utils/ansible/">
          <arg>--inventory-file=production</arg>
          <arg>web</arg>
          <arg>-m</arg>
          <arg>apt</arg>
          <arg>-a</arg>
  <arg>name=python-matheval state=latest update_cache=yes</arg>
          <runif status="passed" />
        </exec>
      </tasks>
    </job>
  </jobs>
</stage>
Listing 10-4

GoCD Configuration for Distributing in, and Deploying to, the Production Environment

The only real news here is the second line
<approval type="manual" />

which makes GoCD proceed to this stage only when someone clicks the approval arrow in the web interface.

You also must fill out the inventory file called production with the list of your server or servers.

10.5 Achievement Unlocked: Basic Continuous Delivery

To recapitulate, the pipeline
  • Is triggered automatically from commits in the source code

  • Automatically builds a Debian package from each commit

  • Uploads it to a repository for the testing environment

  • Automatically installs it in the testing environment

  • Uploads it, upon manual approval, to a repository for the production environment

  • Automatically installs the new version in production

The basic framework for automated deployments from a Git commit in the sources to software running in production is now in place.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset