Moritz Lenz
Python Continuous Integration and DeliveryA Concise Guide with Examples
Moritz Lenz
Fürth, Bayern, Germany
ISBN 978-1-4842-4280-3e-ISBN 978-1-4842-4281-0
https://doi.org/10.1007/978-1-4842-4281-0
Library of Congress Control Number: 2018967720
© Moritz Lenz 2019
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image, we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication, neither the author nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected], or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science+Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
Introduction

One of the keys to successful software development is getting fast feedback. This helps developers avoid going down blind alleys, and in the case of a bug that is revealed quickly, it can be fixed while the code is still fresh in the developer’s mind.

On the business side, fast feedback also helps the stakeholders and the product manager not to build features that turn out not to be useful, thus avoiding wasted effort. Achieving a mutual understanding about the desired product is a very difficult problem in any software project. Showing a working (if partial) product early on often helps to eliminate misunderstandings between stakeholders and developers.

There are many ways to add feedback loops on different levels, from adding linting and other code checks in the IDE to agile processes that emphasize incremental value delivery. The first part of this book focuses on software tests and their automatic execution, a practice known as continuous integration (CI).

When implementing CI, you set up a server that automatically tests every change to the source code, potentially in multiple environments, such as on combinations of operating system and programming language versions.

The next logical step, and the topic of the second part of this book, is continuous delivery (CD). After building and testing code, you add more steps to the automated process: automated deployment to one or more test environments, more tests in the installed state, and, finally, deployment to a production environment. The last step is typically guarded by a manual approval gate.

CD extends the automation and, thus, the possibility for quick iteration cycles, all the way into the production environment, where the software can deliver value for you. With such a mechanism in place, you can quickly obtain feedback or usage data from real customers and assess whether expanding on a feature is useful or discover bugs while the developers still remember the code that they wrote.

The code examples in this book use Python. Owing to its dynamic nature, Python is well-suited to small experiments and fast feedback. The well-stocked standard library and vast ecosystem of available libraries and frameworks, as well as Python’s clean syntax, make it a good choice, even for larger applications. Python is commonly used in many domains, for example, web development, data science and machine learning, Internet of things (IoT), and system automation. It is becoming the lingua franca of professional programmers and those who just touch a subject to automate some part of their job or hobby.

Python comes in two major language versions, 2 and 3. Because Python 2 support is scheduled to end in 2020, and nearly all major libraries now support Python 3, new projects should be started in Python 3, and legacy applications should be ported to that language version as well, if possible. Hence, this book assumes that “Python” refers to Python 3, unless explicitly stated otherwise. If you only know Python 2, rest assured that you will easily understand the source code contained in this book, and transferring the knowledge to Python 3 is very easy.

I. 1 Intended Audience

This book is targeted at technical people involved in the software-delivery process: software developers, architects, release engineers, and DevOps engineers.

The chapters that use source code examples assume basic familiarity with the Python programming language. If you are familiar with other programming languages, spending a few hours reading introductory material on Python will likely bring you to a level at which you can easily follow the code examples in this book.

The sample infrastructure uses Debian GNU/Linux, so familiarity with that operating system is helpful, though not required.

I. 2 Code Examples

Code examples used in this book are available on GitHub under the python-ci-cd organization at https://github.com/python-ci-cd or via the Download Source Code link located at www.apress.com/9781484242803 .

Because some examples rely on automatically fetching code from certain Git repositories, a split into several repositories is necessary. Several chapters reference individual repositories under this namespace.

Acknowledgments

Writing a book is not a solitary endeavor and is only possible with help from many individuals and organizations. I would like to thank all of my beta readers who provided feedback. These include, in no particular order, Karl Vogel, Mikhail Itkin, Carl Mäsak, Martin Thurn, Shlomi Fish, Richard Lippmann, Richard Foley, and Roman Filippov. Paul Cochrane deserves special thanks for reviewing and providing feedback on the blog posts and manuscript and for being available to discuss content, ideas, and organization matters.

I also want to thank my publishing team at Apress: Steve Anglin, Mark Powers, and Matthew Moodie, as well as everybody doing awesome work in the background, such as cover design, typesetting, and marketing.

Finally, thanks go to my parents, for kindling both my love for books and engineering. And most important, to my family: to Signe, my wife, for constant support; and to my daughters, Ida and Ronja, for keeping me grounded in the real world and bringing joy to my life.

Table of Contents

Chapter 1:​ Automated Testing 1
1.​1 What Do We Want from a Test?​ 1
Fast Feedback 1
Confidence 2
Debugging Aid 3
Design Help 3
Specification of the Product 4
1.​2 Downsides of Tests 5
Effort 5
Extra Code to Maintain 5
Brittleness 6
False Sense of Security 6
1.​3 Characteristics of a Good Test 7
1.​4 Kinds of Tests 7
Unit Tests 7
Integration Tests 8
System Tests 9
Smoke Tests 10
Performance Tests 10
1.​5 Summary 12
Chapter 2 :​ Unit Testing in Python 13
2.​1 Digression:​ Virtualenvs 14
2.​2 Getting Started with Unit Tests 15
The First Test 16
Writing More Tests 17
Testing the Unhappy Path 19
2.​3 Dealing with Dependencies 20
Separating Logic from External Dependencies 21
Dependency Injection for Testing 24
Mock Objects 27
Patching 28
2.​4 Separating Code and Tests 30
Tweaking the Python Path 31
2.​5 More on Unit Testing and Pytest 32
2.​6 Running Unit Tests in Fresh Environments 32
2.​7 Another Sample Project:​ matheval 34
Application Logic 34
2.​8 Summary 38
Chapter 3 :​ Continuous Integration with Jenkins 39
3.​1 Continuous Integration Servers 40
3.​2 Getting Started with Jenkins 41
Run Jenkins in Docker 41
Configure a Source Code Repository 43
Creating the First Jenkins Job 43
3.​3 Exporting More Test Details to Jenkins 47
3.​4 Patterns for Working with Jenkins 48
Responsibilities​ 49
Notifications 49
Feature Branches and Pull Requests 50
3.​5 Other Metrics in Jenkins 50
Code Coverage 51
Complexity 51
Coding Style 51
Architectural Constraint Checking 52
3.​6 Summary 52
Chapter 4 :​ Continuous Delivery 53
4.​1 Reasons for CD and Automated Deployments 54
Time Savings 54
Shorter Release Cycles 55
Shorter Feedback Cycles 55
Reliability of Releases 56
Smaller Increments Make Triaging Easier 57
More Architectural Freedom 57
Advanced Quality Assurance Techniques 58
4.​2 A Plan for CD 59
The Pipeline Architecture 59
Anti-Pattern:​ Separate Builds per Environment 61
Everything Hinges on the Packaging Format 62
Technology for Managing Debian Repositories 63
Tooling for Installing Packages 64
Controlling the Pipeline 65
4.​3 Summary 66
Chapter 5 :​ Building Packages 67
5.​1 Creating a Python Source Tarball 67
5.​2 Debian Packaging with dh-virtualenv 68
Getting Started with Packaging 69
5.​3 The control File 70
Directing the Build Process 70
Declaring Python Dependencies 71
Building the Package 72
Creating the python-matheval Package 72
Tradeoffs of dh-virtualenv 74
5.​4 Summary 75
Chapter 6 :​ Distributing Debian Packages 77
6.​1 Signatures 77
6.​2 Preparing the Repository 78
6.​3 Automating Repository Creation and Package Addition 80
6.​4 Serving the Repositories 82
Configuring a Machine to Use the Repository 84
6.​5 Summary 85
Chapter 7 :​ Package Deployment 87
7.​1 Ansible:​ A Primer 87
Connections and Inventory 88
Modules 90
The shell Module 90
The copy Module 91
The template Module 92
The file Module 93
The apt Module 93
The yum and zypper Modules 94
The package Module 94
Application-Specific Modules 95
Playbooks 95
Variables 99
Roles 101
7.​2 Deploying with Ansible 104
7.​3 Summary 106
Chapter 8 :​ A Virtual Playground for Automating Deployments 107
8.​1 Requirements and Resource Usage 107
8.​2 Introducing Vagrant 108
Network and Vagrant Setup 110
8.​3 Configuring the Machines 113
8.​4 Summary 121
Chapter 9 :​ Building in the Pipeline with Go Continuous Delivery 123
9.​1 About Go Continuous Delivery 123
Pipeline Organization 124
Matching of Jobs to Agents 125
A Word on Environments 126
Materials 126
Artifacts 127
9.​2 Installation 127
Installing the GoCD Server on Debian 127
Installing a GoCD Agent on Debian 129
First Contact with GoCD’s XML Configuration 130
Creating an SSH Key 132
9.​3 Building in the Pipeline 132
Directory Layout 134
Stages, Jobs, Tasks, and Artifacts 134
The Pipeline in Action 136
Version Recycling Considered Harmful 137
Constructing Unique Version Numbers 137
Other Bits and Pieces Around the Build 139
Plugging It into GoCD 139
9.​4 Summary 141
Chapter 10 :​ Distributing and Deploying Packages in the Pipeline 143
10.​1 Uploading in the Pipeline 143
User Accounts and Security 146
10.​2 Deploying in the Pipeline 147
10.​3 Results 149
10.​4 Going All the Way to Production 150
10.​5 Achievement Unlocked:​ Basic Continuous Delivery 152
Chapter 11 :​ Pipeline Improvements 153
11.​1 Rollbacks and Installing Specific Versions 153
Implementation 154
Try It! 156
11.​2 Running Smoke Tests in the Pipeline 157
When to Smoke?​ 157
White Box Smoke Testing 158
Sample Black Box Smoke Test 158
Adding Smoke Tests to the Pipeline and Rolling Releases 159
11.​3 Configuration Templates 161
11.​4 Avoiding the Rebuild Stampede 164
11.​5 Summary 166
Chapter 12 :​ Security 167
12.​1 The Dangers of Centralization 167
12.​2 Time to Market for Security Fixes 169
12.​3 Audits and Software Bill of Materials 169
12.​4 Summary 170
Chapter 13 :​ State Management 171
13.​1 Synchronization Between Code and Database Versions 172
13.​2 Decoupling Application and Database Versions 173
Example of a Schema Change 174
Prerequisites 179
Tooling 180
Structure 180
No Silver Bullet 181
13.​3 Summary 181
Chapter 14 :​ Conclusions and Outlook 183
14.​1 What’s Next?​ 183
Improved Quality Assurance 183
Metrics 184
Infrastructure Automation 185
14.​2 Conclusions 187
Index 189

About the Author and About the Technical Reviewer

About the Author

Moritz Lenz
../images/456760_1_En_BookFrontmatter_Figb_HTML.jpg

is a prolific blogger, author, and contributor to Open Source projects.

He works as software architect and principal software engineer for a midsize IT outsourcing company, where he has created a Continuous Integration and Delivery system for over 50 software libraries and applications.

 

About the Technical Reviewer

Michael Thomas
../images/456760_1_En_BookFrontmatter_Figc_HTML.jpg

has worked in software development for over 20 years as an individual contributor, team lead, program manager, and vice president of engineering. Michael has more than 10 years of experience working with mobile devices. His current focus is in the medical sector, using mobile devices to accelerate information transfer between patients and health-care providers.

 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset