Once you have automated tests for your software, you must take care to keep those tests passing. With changes to the code or to infrastructure, or with new library versions, tests can start failing.
If you let them fail and don’t do anything against this creeping entropy, the diagnostic value of the tests starts to drop, and new regressions tend to be covered up by the general noise. Keeping your tests passing, and continuously checking in new features and bug fixes, is a practice that must be part of the engineering culture of a software development team.
There are tools that can help the team. Continuous integration (CI) servers monitor version control repository and automatically run test suites on each new commit, possibly on a wide variety of platforms. They can notify developers when they’ve caused some tests to fail, give an overview of the history of a test job, and visualize trend data, such as test coverage.
When you use such a tool, it helps you to discover when tests start to fail, triage the failure to certain commits or platforms, and render the “it works on my machine” mantra obsolete, by providing an independent assessment. The engineers working on the software, however, still require the discipline to fix the test failures that a CI tool discovers.
3.1 Continuous Integration Servers
There are two kinds of CI servers, based on their deployment model. You install and run on-premise software on your own infrastructure, while cloud-based or software as a service (SaaS) software is typically run by the vendor that creates the CI server.
In enterprise settings, on-premise software tends to be the preferred solution, because it means the source code that is being tested doesn’t have to leave the organization’s network.
The most popular open source, on-premise CI server software is Jenkins,1 a Java-based project under the MIT license, which we will use later in this chapter. Other examples in this category include Buildbot2 (written in Python) and CruiseControl.3 Popular closed source, on-premise CI software includes TeamCity4 by JetBrains and Atlassian’s Bamboo.5
In the realm of hosted CI services, Travis CI 6 is very popular, owing to its excellent integration with GitHub. Travis is also open source and can be self-hosted. AppVeyor7 is frequently used for Windows-based CI. Both Travis and AppVeyor offer free plans for open source projects.
Most CI software has a central server component that can poll source code repositories for changes and that can also be triggered by hooks. If a change in the source code repository is detected, this triggers a job. The job can either be configured centrally in the server or in the source code repository. For example, Travis expects a file called .travis.yml, in the root of the repository that instructs Travis on how to prepare the environment and which commands to execute, to trigger the build and test.
Once the CI server knows which tests to execute, and in which environments, it typically delegates the actual test runs to worker nodes. The worker nodes then report their results back to the server, which takes care of sending notifications and making the outputs and results available for inspection through a web interface.
3.2 Getting Started with Jenkins
First, you need a working Jenkins installation. The official web site8 contains instructions on how to install and set up Jenkins on all common operating systems. Following, you can also find quick instructions to get a Docker-based Jenkins playground running.
Run Jenkins in Docker
Usually, in a production environment, you’d run the Jenkins server on one machine and have several build workers on different (virtual) machines. For the sake of easy setup, we’ll forego this sensible distinction and run the server and all build jobs within the same Docker container, just to have fewer docker containers to manage.
To do this, we use the official Docker image from Jenkins but add the tox Python module (which we will use to create reproducible build environments), as well as the Python version we want to test under.
This first downloads the image jenkins/jenkins:lts from Dockerhub, which might take a few minutes. Then it runs the commands from the RUN lines of the Dockerfile, which installs pip and then tox. The resulting image gets the name jenkins-python.
The -v ... argument attaches a volume, which makes the Jenkins server not lose state when the container is killed and restarted.
Copy the password, then point your browser to http://127.0.0.1:8080/ and follow the setup instructions (which require the password as the first step). When it comes to plug-ins, add the Python Plugin to the list of plug-ins to be installed.
The plug-in installation process can again take a few minutes. After that, you have a working Jenkins server.
Configure a Source Code Repository
Jenkins runs jobs based on source code in a source control repository. For a proper software development project, you likely already have a place where you store the code. If not, you can use one of the many cloud hosting services, such as GitHub,9 GitLab,10 or Atlassian’s Bitbucket.11 You can also install GitLab, Gitea,12 Gogs,13 or other Git management projects on your own infrastructure.
In either case, you end up with a Git repository that is reachable through the network, which is exactly what Jenkins needs. For the sake of demonstration, I’ve created a public GitHub repository at https://github.com/python-ci-cd/python-webcount .
In the case of private repositories, you also need either an SSH key pair or a combination of username and password, to access the repository.
Creating the First Jenkins Job
We want Jenkins to regularly run the tests of our project. To this end, we need to create a job, which configures all of the details on where and how Jenkins gets the source code and runs the tests.
To create a job, click the New Item link in the left column of the Jenkins starting page. Then, you have to enter a name, for example, the name of the repository, python-webcount, and a job type, here Multi-configuration project. Then click OK to proceed.
Select Git in the section Source Code Management, and enter the repository URL (for example, https://github.com/python-ci-cd/python-webcount.git ). For private repositories, you must also enter valid credentials below the URL (Figure 3-1).
In the Build Trigger section, select Poll SCM and enter the string H/5 * * * * as the schedule, which means polling every five minutes.
Under Configuration Matrix, add a User-defined Axis with name TOXENV and value py35. If you have more Python versions installed in Jenkins and defined in the project’s tox.ini file, you can add them here, separated by spaces (Figure 3-2).
In the Build section, select Execute Python script and paste the following short Python script into the script area (Figure 3-3).
Jenkins configuration: Source Code Management
Jenkins configuration: Build Triggers and Configuration Matrix
Jenkins configuration: Build configuration
When you’ve added these pieces of information, you can save the page and have a first working CI job.
Every five minutes, Jenkins will now check the Git repository for new commits, and if there are any, it will fetch them, run the tests through tox, and make the status available in the front end.
When you define more tox environments, Jenkins shows you whether the test passes or fails per environment and gives you a history for each environment.
3.3 Exporting More Test Details to Jenkins
In its current state, Jenkins detects test status purely based on the exit code of the script that runs, which doesn’t provide good granularity. We can improve the granularity by instructing tox to write a machine-readable summary and getting Jenkins to read this data.
For the environment py35, pytest then creates a file junit-py35.xml that describes the test run in more detail.
Post-build action: Publish JUnit test result report
When the job runs again, Jenkins picks up the status of individual test functions and even reports runtimes for each function. This allows a much better diagnostic directly from the Jenkins web interface.
3.4 Patterns for Working with Jenkins
Now that the basics for testing with Jenkins are in place, it’s time to think about how you would actually work with it on a day-to-day basis. Most of that is centered on keeping the tests green, that is, all the tests passing. Experience shows that if you don’t focus on keeping your jobs green, developers get used to having failing tests and then slip from just 1% failing tests to the test runs becoming pure noise, hence losing their value.
In addition, you should make a review of the tests part of your development workflow, to ensure that tests accurately reflect the requirements, even when a new feature changed the requirements.
Responsibilities
If several developers work on the same code base, it is essential to define clear responsibilities for passing the test suite. Typically, the one who breaks the test suite, as measured by going from green to red in Jenkins, is responsible for fixing it again.
In a team that works like a well-oiled machine, that one rule might be enough. If that’s not the case, it can make sense to appoint a build master who takes on the primary responsibility for a green test suite.
This doesn’t mean that the build master has to clean up all the failing tests. It is more of a managerial role of talking to those who broke the test suite and making sure they clean up after themselves. If that doesn’t turn out to be practical, revert the commits that caused trouble and schedule it for re-inclusion when it passes all tests.
The build master role can also rotate between different developers, if nobody feels the calling of always doing it.
Notifications
Notifications can help a development team to keep the tests green, simply by informing the members about broken tests, thus letting them know that an action is required. The notification can be by e-mail, to a chat system that the developers use, or even to a monitor that is physically present in the developer’s office. Jenkins’s rich ecosystem of plug-ins cover nearly all notification technologies commonly in use.
If you configure Jenkins to send notifications when test suites break, also configure it to send notifications when it passes again. Otherwise, everybody involved will learn to hate notifications from Jenkins giving only negative feedback, which is not a good position for a successful CI process.
Feature Branches and Pull Requests
If your development workflow is based on feature branches and, possibly, merge requests or pull requests (wherein a second person reviews and merges the changes), it makes sense to cover those branches in your CI system as well. The developer who is in charge of merging the branch can then do so in the knowledge that all tests still pass on the feature branch.
In the case of formal merge requests or pull requests, Git hosting solutions such as GitHub and GitLab even support a mode in which the request can only be merged if all tests are passing. In such a scenario, it makes sense to test not just the feature branch but the result of a merge between the feature branch and the development branch. This avoids the situation in which all tests pass both in the development branch and in the feature branch, but the merge breaks some tests.
Such integrations are available for Jenkins as plug-ins.14
3.5 Other Metrics in Jenkins
Once a team works smoothly with a CI system, you can use it to gather other metrics about the software and steer it into a desirable direction. Be careful to introduce such metrics only in limited experiments and expand them to larger projects only if you find that they provide tangible value to the development process. They all come with the cost of maintenance and of reducing the developer’s autonomy.
Code Coverage
Code coverage measures the percentage of statements or expressions in a piece of source code that is executed during test runs, compared to the total number of expressions. Code coverage serves as a simple proxy for how thorough a test suite exercises the code, though it should be taken with a grain of salt, because combinatoric explosion of path numbers through a piece of code can lead to undetected bugs, even in tested code.
The pytest-cov15 project gathers such data, and you can even use it to make your CI jobs fail, if the test coverage falls below a certain threshold.
Complexity
There are various attempts at measuring the complexity of a code base, for example, cyclomatic complexity and maintainability index, which a tool called radon16 can calculate for Python code. While those numbers aren’t too reliable, observing their trend can give you some insight into the health of a code base.
Coding Style
When a project defines a coding style, it can use a tool like pylint17 or Flake818 to check that the code in the repository actually adheres to the guidelines and even fail the build if violations are detected. These two tools come with a set of default rules but can be customized to your own rules.
Architectural Constraint Checking
UI may not use the storage back end directly, only business logic.
The storage back end may not use the UI directly.
If those layers are handled as modules in Python code, you can write a small script that analyzes the import statements in all source files and checks if any of them violate these rules. A static import analyzer such as snakefood19 can make this easier.
Such a tool should make the CI step fail when a violation is detected. This allows you to track whether the ideas of an architecture are actually implemented in the code and prevent the code from slowly weakening the underlying architecture principles.
3.6 Summary
Jenkins is a CI server that automatically runs test suites for you, usually for every new commit in a source repository. This gives you an objective view of the state of your test suite, possibly on multiple Python versions or platforms.
Once you have this view, you can have a process whereby the test suite is always kept passing, and you can derive value from the test suite.
When a mature team works well with a CI process, you can introduce other metrics, such as code coverage or adherence to architectural rules, into the CI process.