Chapter 11: Trunk-Based Development

One of the capabilities that are highly correlated with accelerated engineering velocity is trunk-based development (also known as TBD). High-performing teams have fewer than three active branches at any time, and their branches have a short lifetime (less than a day) before being merged into the main branch (Forsgren N., Humble, J., and Kim, G. 2018, page 98). Unfortunately, TBD is not a git workflow but rather a branching model of choice that has been in use since the 80s. It is not well defined and leaves a lot of room for interpretation, especially when it comes to using it with GitHub. Also, I personally find that only moving to a trunk-based workflow does not increase the performance too much. Only large teams with a highly complex workflow that are already stuck in merge hell really have this high impact. For most teams, it is more a combination of different capabilities such as feature flags and continuous integration/continuous deployment (CI/CD), together with a trunk-based workflow, that makes a big difference.

In this chapter, I'll explain the benefits of trunk-based workflows. We'll also cover their difference from other branching workflows, and I'll introduce you to what I believe is the best git workflow to accelerate your software delivery.

The chapter covers the following topics:

  • Trunk-based development
  • Why you should avoid complex branching
  • Other git workflows
  • Accelerating with MyFlow
  • Case study

Trunk-based development

Trunk-based development is a source-control branching model, where developers merge small and frequent updates to a single branch (often called a trunk, but in git, this is commonly referred to as the main branch) and resist any pressure to create other long-lived development branches (see https://trunkbaseddevelopment.com).

The base idea is that the main branch is always in a clean state so that any developer, at any time, can create a new branch based upon the main branch that builds successfully.

To keep the branch in a clean state, developers must take multiple measures to ensure only code that does not break anything is merged back to the main branch, as outlined here:

  • Fetch the newest changes from the main branch
  • Perform a clean test
  • Run all tests
  • Have high cohesion with your team (pair programming or code review)

As you can see, this is predestined for a protected main branch and pull requests (PRs) with a CI build that has a PR trigger and builds and tests your changes. However, some teams prefer to do these steps manually and directly push to main without branch protection. In small, highly cohesive, and co-located teams that practice pair-programming, this can be very effective, but it takes a lot of discipline. In complex environments or in distributed teams that work asynchronously, I would always recommend using branch protection and PRs.

Why you should avoid complex branching

When we talk about branches, we often use the terms long-lived and short-lived, which refer to time. I find this somehow misleading. Branches are about changes, and changes can hardly be measured in time. Developers can write 8 hours of code with a lot of refactoring and try to merge that very complex branch in 1 day. This would still be considered short-lived if they measured it in time only. Conversely, if they have a branch with just one line changed—for example, the update of a package that the code depends on—but the branch stays open for 3 weeks as the team must solve some architectural questions regarding the change, from a time perspective it would be long-lived, even if it would be very simple to rebase the changes on top of main.

Time doesn't seem to be the best measure to distinguish good and bad practices for branches; it is a combination of complexity and time.

The more changes that happen in the base branch from which you created your branch until you try to merge your changes back, the harder it is to merge these changes with changes in your branch. The complexity can come from one very complex merge, or from many developers merging many small changes. To avoid merging, many teams try to finish work on a feature before merging back. This, of course, leads to more complex changes that then make it difficult for the other features to merge—so-called merge hell—whereby before releasing, all the features have to be integrated into the new release.

To avoid merge hell, you should pull the latest version of the main branch regularly. As long as you can merge or rebase without problems, the integration of your branch is not a problem, but if your changes get too complex, it is a problem for the other developers, as they will probably have a problem if you merge your changes back. That's why you should merge your changes before they exceed a certain amount of complexity. The extent of the complexity depends a lot on the code you modify and you would need to consider the following points:

  • Are you working with existing code or new code?
  • Is the code complex with a lot of dependencies, or is it simple code?
  • Are you working with isolated code or code with high cohesion?
  • How many people change the code at the same time?
  • Is there refactoring of a lot of code at the same time?

I believe this is why people tend to use time as a measure instead of complexity—there is just no good measure for complexity. So, as a rule of thumb: if you work on a more complex feature, you should at least merge your changes back to the main branch once a day, but if your changes are simple, there is no problem leaving your branch/PR open for a longer time. Remember that it is not about time but complexity!

Other git workflows

Before we have a closer look into what I believe to be the most effective git workflow for DevOps teams using GitHub, I want to make an introduction to the most popular workflows.

Gitflow

Gitflow is still one of the most popular workflows. It was introduced in 2010 by Vincent Driessen (see https://nvie.com/posts/a-successful-git-branching-model/) and became very popular. Gitflow has a nice poster, and it is a very descriptive introduction on how to solve problems in git such as releasing using tags and working with branches that get deleted after they have been merged (see Figure 11.1):

Figure 11.1 – Gitflow overview

Figure 11.1 – Gitflow overview

Gitflow is great if you ship your software every few months to different customers, want to bundle some features to a new major version that is licensed separately, and have to maintain many versions for many years. In 2010, this was the common release flow for nearly all software, but in complex environments, the workflow raises some problems. The workflow is not trunk-based and has multiple long-lived branches. The integration between these branches can lead to merge hell in complex environments. With the rise of DevOps and CI/CD practices, the workflow got a bad reputation.

If you want to accelerate your software delivery with DevOps, Gitflow is not the right branching workflow for you! But many of the concepts can be found in other workflows.

GitHub flow

GitHub flow focuses a lot on collaboration with PRs. You create a branch with a descriptive name and make your first changes. Then, you create a PR and collaborate with your reviewers through comments on the code. Once the PR is ready, it gets shipped to production before merging it to the main branch (see Figure 11.2)

Figure 11.2 – GitHub flow

Figure 11.2 – GitHub flow

GitHub flow is trunk-based, and it is very popular. The basic part—without the deployment of PRs—is the base for most other workflows. The problem is the deployment. Deploying each PR to production creates a bottleneck and does not scale very well. GitHub itself uses ChatOps and deploy trains to solve that issue (Aman Gupta, 2015), but this seems a bit of overkill to me. The idea that only changes that have proven to work in production are merged to main is compelling, but it is a goal that basically cannot be reached in complex environments. You would need quite some time to see the changes work isolated in production to really be sure that they did not break anything, but with that time, the bottleneck prevents other teams or team members to merge their changes. I think that in a DevOps world with the principles of fail fast and roll forward, it's best to validate PRs in an isolated environment and deploy them to production after you have merged the PR using the push trigger of your main branch. If the changes break production, you still can deploy the last version that worked (roll back), or you fix the error and deploy the fix right away (roll forward). You don't need a clean main branch to perform either of these options.

Another thing I dislike about GitHub flow is that it is not very explicit about the number of users, branches, and PRs. A feature branch might imply that multiple persons commit to the same feature branch. I don't see this happen often, but just from the documentation, it is not unambiguous.

Release flow

Release flow is based upon GitHub flow, but instead of deploying PRs continuously, it adds one-way release branches. The branches do not get merged back, and bug fixes follow the upstream-first principle: they get fixed in a branch of main, and changes get cherry-picked into a branch of the release branch (Edward Thomson, 2018). This way, it is impossible to forget to apply a bug fix to main (see Figure 11.3):

Figure 11.3 – Release flow

Figure 11.3 – Release flow

Release flow is not CD! Creating releases is still a process that has to be triggered separately. If you have to maintain different versions of your software, release flow is a good way to do it. But if you can, you should try to achieve CD.

GitLab flow

GitLab flow is also based upon GitHub flow. It adds environment branches (such as development, staging, pre-production, and production), and each deployment happens on a merge to these environments (see Figure 11.4):

Figure 11.4 – GitLab environment branches

Figure 11.4 – GitLab environment branches

Since the changes only flow downstream, you can be sure that all changes are tested in all environments. GitLab flow also follows the upstream-first principle. If you find a bug in one of the environments, you create a feature branch of main and cherry-pick changes to all environments. Bug fixing works in GitLab flow the same way it works in release flow.

If you don't have pipelines that support multiple environments—such as GitHub Actions—GitLab flow might provide a nice way to automate your approvals and deployments for your environments. Personally, I don't see the value in having code for environments separated if you perform bug fixes upstream anyway. I prefer to build the code once and then deploy the output to all environments in a sequence. But there might be situations in which this workflow makes sense—for example, for static websites that deploy directly from the repository.

Accelerating with MyFlow

As you can see, git workflows are just a collection of solutions for different use cases. The main difference is in whether they are trunk-based or not and if they are explicit about some things or not. As I find all workflows lacking something, I created my own workflow: MyFlow.

MyFlow is a lightweight, trunk-based workflow based on PRs. MyFlow is not a new invention! Many teams already work this way. It is a very natural way to branch and merge if you focus on collaboration with PRs. I just gave it a name, and I can see people picking it up easily.

The main branch

Since MyFlow is trunk-based, there is only one main branch called main, and it should always be in a clean state. The main branch should always build, and it should be possible to release it to production at any time. That's why you should protect main with a branch protection rule. A good branch protection rule would include at least the following criteria:

  • Require a minimum of two PR reviews before merging
  • Dismiss stale PR approvals when new commits are pushed
  • Require reviews from code owners
  • Require status checks to pass before merging that includes your CI build, test execution, code analysis, and linters
  • Include administrators in restrictions
  • Permit force pushes

The more you automate using the CI build, the more likely you can keep your branch in a clean state.

All other branches are always branched off main. Since this is the default branch, you never have to specify a source branch when you create a new one. This simplifies things and removes a source of error.

Private topic branches

Figure 11.5 shows the basic concept of MyFlow:

Figure 11.5 – Basics of MyFlow

Figure 11.5 – Basics of MyFlow

Private topic branches can be used to work on new features, documentation, bugs, infrastructure, and everything else that is in your repository. They are private, which means they only belong to one specific user. Other team members can check out the branch to test the solution, but they are not allowed to directly push changes to this branch. Instead, they must use suggestions in PRs to suggest changes to the author of the PR.

To indicate that branches are private, I recommend a naming convention such as users/* or private/* that makes this obvious. I also recommend including the identifier (ID) of the issue or bug in the name. This makes it easy to reference it later in the commit message. A good convention would look like this:

users/<username>/<id>_<topic>

To start working on a new topic, you create a new local branch, as follows:

$ git switch -c <branch> main

You can see an example here:

$ git switch -c users/kaufm/42_new-feature main

> Switched to a new branch 'users/kaufm/42_new-feature'

Create your first modifications and commit and push them to the server. It does not matter what you modify—you could just add a blank to a file. You can overwrite it later anyway. You can see an example here:

$ git add .

$ git commit

$ git push --set-upstream origin <branch>

Now, here's the preceding example with further information:

$ git add .

$ git commit -m "New feature #42"

$ git push --set-upstream origin users/kaufm/42_new-feature

Note

Note that I use GitHub command-line interface (GitHub CLI) (https://cli.github.com/) to interact with PRs as I find it easier to read and understand than to use screenshots of the web user interface (UI). You can do the same using the web UI.

Create a PR and mark it as draft, as follows:

$ gh pr create --fill --draft

This way, the team knows that you are working on that topic. A quick view of a list of open PRs should give you a nice overview of the topics the team currently is working on.

Note

You can omit the -m argument when committing changes and add a multiline commit message in your default editor. The first line will be the title of the PR; the rest of the message will be the body. You could also set title (--title or -t) and body (--body or -b) when creating a PR instead of --fill.

You can now start working on your topic, and you can use the full power of git. If you want to add changes to your previous commit, for example, you can do so with the --amend option, as follows:

$ git commit --amend

Or, if you want to combine the last three commits into one single commit, you can run the following command:

$ git reset --soft HEAD~3

$ git commit

If you want to merge all commits in a branch into one commit, you can run the following command:

$ git reset --soft main

$ git commit

Or, if you want complete freedom to rearrange and squash all your commits, you can use interactive rebase, like this:

$ git rebase -i main

To push changes to the server, you use the following command:

$ git push origin +<branch>

Here's the preceding example with the branch name populated:

$ git push origin +users/kaufm/42_new-feature

Note the + plus sign before the branch name. This causes a force push, but only to a specific branch. If you are not messing with your branch history, you can perform a normal git push operation, and if your branches are well protected and you know what you are doing, a normal force push might be more convenient, as illustrated here:

$ git push -f

If you already want help or the opinions of teammates on your code, you can mention them in comments in the PR. If they want to propose changes, they use the suggestions feature in PR comments. This way, you apply the changes, and you can make sure that you have a clean state in your repository before doing so.

Whenever you feel your work is ready, you change the state of your PR from draft to ready and activate auto-merge, as follows:

$ gh pr ready

$ gh pr merge --auto --delete-branch --rebase

Note

Note that I specified --rebase as the merge method. This is a good merge strategy for small teams that like to craft a good and concise commit history. If you prefer --squash or --merge, adjust your merge strategy accordingly.

Your reviewers can still create suggestions in their comments, and you can keep collaborating. But once all approvals and all automated checks have completed, the PR will be merged automatically and the branch gets deleted. Automated checks run on the pull_request trigger and can include installing the application in an isolated environment and running all sorts of tests.

If your PR has been merged and the branch has been deleted, you clean up your local environment, like this:

$ git switch main

$ git pull --prune

This will change your current branch to main, pull the changed branch from the server, and delete local branches that have been deleted on the server.

Releasing

Once your changes are merged to main, the push trigger on main will start the deployment to production, independent of whether you use environments or a ring-based approach.

If you have to maintain multiple versions, you can use tags together with GitHub releases (as I showed you in Chapter 8, Managing Dependencies Using GitHub Packages). Use the release trigger in a workflow and deploy the application, and use GitVersion to automatically generate your version numbers, as illustrated here:

$ gh release create <tag> --notes "<release notes>"

Here's an example of this:

$ gh release create v1.1 --notes "Added new feature"

You can also take advantage of the autogeneration of release notes. Unfortunately, this feature is not yet available through the CLI. You must create your release using the UI for that to work.

As we fix bugs following the upstream-first principle anyway, there is no real benefit in creating a release branch for every release if we don't have to perform a hotfix. The tag that is generated when you create your release works just fine.

Hotfix

If you have to provide a hotfix for older releases, you can check out the tag and create a new hotfix branch, like this:

$ git switch -c <hotfix-branch> <tag>

$ git push --set-upstream origin <branch>

Here's an example of this:

$ git switch -c hotfix/v1.1.1 v1.1

$ git push --set-upstream origin hotfix/1.1.1

Now, switch back to main and fix the bug in a normal topic branch (for example, users/kaufm/666_fix-bug). Now, cherry-pick the commit with the fix to the hotfix branch, as follows:

$ git switch <hotfix-branch>

$ git cherry-pick <commit SHA>

$ git push

You can use the secure hash algorithm (SHA) of the commit you want to cherry-pick. Or you can use the name of the branch if the commit is the tip of the branch, as follows:

$ git switch hotfix/v1.1.1

$ git cherry-pick users/kaufm/42_fix-bug

$ git push

This will cherry-pick the tip of the topic branch. Figure 11.6 shows how a hotfix for an older release works:

Figure 11.6 – Performing hotfixes on older releases

Figure 11.6 – Performing hotfixes on older releases

You could also merge the fix to main first and then cherry-pick the commit from there. This ensures that the code adheres to all your branch policies.

You could also cherry-pick into a temporary branch based on the hotfix branch and merge the cherry-picked fix using another PR. This depends on how complex your environment is and how big the differences between the main and hotfix branches are.

Automation

If you have a workflow with naming conventions, there are certain sequences of commands that you use very often. To reduce typos and simplify your workflow, you can automate these using git aliases. The best way to do this is to edit your .gitconfig file in the editor of your choice, like this:

$ git config --global --edit

Add a section, [alias], if it does not exist yet and add an alias, like this:

[alias]
    mfstart = "!f() { 
        git switch -c users/$1/$2_$3 && 
        git commit && 
        git push --set-upstream origin users/$1/$2_$3 && 
        gh pr create --fill --draft; 
    };f"

This alias is called mfstart and would be called specifying the username, issue ID, and topic, as illustrated here:

$ git mfstart kaufm 42 new-feature

It switches to a new branch and commits the current changes in the index, pushes them to the server, and creates a PR.

You can reference individual arguments ($1, $2, …) or all arguments using $@. If you want to chain commands independent of the exit code, you must terminate a command using ;. If you want the next command only to execute if the first one was successful, you can use &&. Note that you have to end each line with a backslash (). This is also the character you use to escape quotation marks.

You can add if statements to branch your logic, like so:

mfrelease = "!f() { 
    if [[ -z "$1" ]]; then 
        echo Please specify a name for the tag; 
    else 
        gh release create $1 --notes $2; 
    fi; 
};f"

Or, you can store values in variables to use them later, as in this example—the current name of the branch your head (HEAD) points to:

mfhotfix = "!f() { 
    head=$(git symbolic-ref HEAD --short); 
    echo Cherry-pick $head onto hotfix/$1 && 
    git switch -c hotfix/$1 && 
    git push --set-upstream origin hotfix/$1 && 
    git cherry-pick $head && 
    git push; 
};f"

These are just examples and the automation depends a lot on the details of the way you work, but it is a very powerful tool and can help you to become more productive.

Case study

With the automation of the release process in place, the two pilot teams have already noted a great boost in productivity. Metrics for lead time and deployment frequency have increased significantly.

The team that used git before they moved from Bitbucket to GitHub followed Gitflow as their branching workflow. Since their web application can be released continuously using their staged deployment workflow, they move to a trunk-based workflow with PR and private branches and deploy after the merge to the main branch using their CI/CD workflow (MyFlow). To integrate often, they decide to use feature flags. As the company needs feature management in the cloud and on-premises, they decide to go with Unleash. The team can use the software-as-a-service (SaaS) service and can start using it right away without having to wait for an on-premises solution.

The second team that migrated from Team Foundation Server (TFS) had been used to a complex branching workflow with a long-living release, service pack, hotfix branches, and a development branch where all features were integrated. As the software is installed on hardware products, multiple releases are stabilized in parallel, and also multiple versions that have to be maintained for years. This means the software cannot be continuously released. The team chooses release flow to manage releases and hotfixes. For development, they also use private branches with PRs and a trunk-based approach. As the products are not connected to the internet, the team relies on their configuration system for feature flags. This technique had been used before to enable the testing of new features on hardware. The team now extends it to integrate changes more frequently.

Summary

git workflows are not so different from each other, and most are built on top of others. It is more important to follow the principles of fail fast and roll forward instead of treating a certain workflow like a dogma. All workflows are just a collection of best practices, and you should only take what you need.

What is important is the size of your changes and the frequency in which you merge them back.

Always follow these rules:

  • Always branch your topic branches of the main branch (trunk-based).
  • If you're working on complex features, make sure to commit at least once per day (using feature flags).
  • If your changes are simple and you only need to change a few lines of code, you can leave your PR open for a longer time. But check that you don't have too many open PRs.

With these rules, the workflow you are actually using is not so important. Pick the things that work for you.

In this chapter, you learned about the benefits of TBD and how you can use it together with git workflows to increase your engineering velocity.

In the next chapter, I will explain how you can use shift-left testing for increased quality and to release with more confidence.

Further reading

You can use the following references from this chapter to get more information on the topics covered:

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset