After reading this book, you should have a solid understanding of how and why to implement continuous integration (CI) and continuous delivery (CD) for a Python project. It is not a small undertaking, but the many examples should get you started pretty quickly, and even an implementation of only some aspects can give you benefits. In a collaborative environment, showing these benefits makes it easier to convince others that it’s worth spending your time on toolchain and process improvements.
14.1 What’s Next?
In this final chapter, let’s look at some concepts that can help you grow an even more mature software development process that ties into CI and CD.
Improved Quality Assurance
Improving the quality of your software can be as simple as increasing the unit test coverage of your application. However, not all classes of errors can be caught that way, for example, performance regressions or errors for cases you didn’t think of before.
To catch performance regressions, you can create a separate performance testing environment and run a predefined set of load and performance tests in this environment. You can add this as another stage to your deployment pipeline.
Handling unexpected cases is harder, because, by definition, they catch you by surprise. For certain types of applications, automatic fuzzing can find inputs that make your application crash and provide these inputs as examples to the developers.
There are architectural approaches to make your applications robust against unexpected user input and error scenarios, but from a tooling perspective, the best you can do is to make the application’s reactions to such errors more robust.
Specialized error trackers can help you to identify such errors. They give developers more insight into how to reproduce and diagnose those errors. For example, Sentry1 is an open source, centralized error tracker with a hosted solution available.
Metrics
In bigger systems and organizations, gathering and aggregating metrics is a requirement for keeping the system manageable. There is even a trend to base monitoring on time-series data.
In the context of a deployment system, some data points you can collect include the start date and duration of each stage or task, which version it built, and the characteristics of that particular version, including performance data, usage data such as engagement rates, size of the generated artifacts, defects and vulnerabilities discovered, and so on.
Making sense of the gathered data is not always easy, and entire books have been written about it. Nonetheless, it’s a good idea to establish a way to collect metrics of all kinds and to create dashboards that help interpret them.
Infrastructure Automation
Configuration management is essential for scaling your infrastructure, but a tool such as Ansible alone isn’t enough for all needs and scales.
Configuration in a Database, Secrets Management
As the amount of configuration data grows, keeping it in plain text files becomes impractical. Therefore, you will have to maintain a database of the configuration and use Ansible’s dynamic inventory mechanism2 to propagate the data into the configuration management system.
However, storing passwords, private keys, and other secrets in a database is always a delicate business. You need an application on top of the database, to avoid leaking such secrets to users who shouldn’t have access to them.
Such applications already exist. Dedicated secret management systems store secrets in encrypted form and carefully control access to them. Examples of such applications are Keywhiz,3 by Square, or Vault,4 by HashiCorp, the authors of Vagrant.
Secret management systems typically offer plug-ins to create service accounts, such as MySQL or PostgreSQL database accounts, and rotate their passwords without human involvement. Crucially, it also means that no human has to ever see the randomly generated passwords.
Instead of pushing application configuration into the machines or containers where the application runs, you can also build your applications to fetch the configuration from a central location. Such a central location is typically called a service discovery system . Tools such as etcd,5 from the CoreOS project, and Consul,6 from HashiCorp, make it easier to manage large amounts of configuration. They also provide additional features, such as basic monitoring for services and exposing only working instances of a service end point to consumers.
To illustrate, consider that an application requiring large amounts of configuration data could be supplied with just a secret key for authentication against the service discovery system and the information about which environment it runs in. The application then reads all of its other configuration from the central service. If the application needs access to a storage service, and there are multiple instances that can provide this service, the monitoring service makes sure that the application gets the address of a working instance.
Such a service discovery approach allows a pattern called immutable infrastructure . This means you build a container (such as a Docker container or even a virtual machine image) once, then, instead of propagating just your application through the various testing environments, you propagate the whole container through them. The cluster management system provides the credentials for connecting to the service discovery system; otherwise, the containers remain unchanged.
Infrastructure As Code
A traditional CD system , as described in the earlier chapters, is usually limited to one branch in a source control system, because there is only one testing environment for deploying the code.
Cloud infrastructure changes the game. It allows declarative descriptions of whole environments, consisting of several databases, services, and virtual servers. Instantiating a new environment then becomes a matter of executing a single command, allowing you to deploy each branch into a new, separate environment.
Leading tools for creating new environments are Terraform7 and CloudFormation.8
14.2 Conclusions
Automating deployments makes both software development and operation more efficient and more pleasant. I’ve shown you a gentle and practical introduction to it, and, in turn, enabled you to introduce CD to your organization.
This is a big step for an organization that develops software, but it is also a small part of automating your infrastructure and a small part of the journey to an efficient and resilient software development process.