Start with Experiments?

If chaos engineering follows the scientific method, then starting with an experiment seems like the right choice. The challenge then quickly becomes, “Which experiment?” Experiments are often complex and detailed, and, more importantly, require a lot of work to plan and execute. That’s a big expenditure of wasted effort should you ultimately decide not to conduct the experiment.

A better approach is to take a step back to figure out what experiments you might find valuable before you go ahead and invest your time and effort.

Instead of starting by injecting failure and chaos, or even by designing full-blown experiments, start by asking “What idea do we want to gather some evidence to support or refute?” or more simply, “What could possibly go wrong?” These questions are the essence of a hypothesis, and so the key to figuring out what experiments to invest in starts with building up a backlog of such questions for your own systems. This collection of questions is called the Hypothesis Backlog.

Introducing Likelihood and Impact

At some point you will begin to exhaust your team’s ideas, or perhaps your list will start to feel endless. A big list of failures is a scary thing and is not very helpful on its own, but with a little more work you can turn it into what you need.

To turn your collection of failures into something more useful you are going to apply a technique called Failure Mode and Effects Analysis Lite. Failure Mode and Effects Analysis is a powerful technique that has long been part of a reliability engineer’s toolbox.⁴ You are going to use a faster version of the technique purely for the refinement of your set of failures, to help you decide which failures should be turned into chaos engineering hypotheses for your backlog.

So far you’ve completed the first two steps of building a Hypothesis Backlog:

• Build a detailed sketch of your system.

• Brainstorm and collect a set of detailed possible system failures across your real system and from the perspective of different attack vectors.

The next step is:

• Understand, discuss, map out, and agree on the relative likelihood and impact of those failures.

Building a Likelihood-Impact Map

Your collection of potential failures needs a little spice to make it really useful. The spice you’ll add in now includes:

• The likelihood that a failure may occur⁵

• The potential impact your system will experience if it does

You can factor in this information to build a likelihood-impact map.

Isn’t Likelihood Just Guesswork?

In many respects, the level of probability that you’ll consider for your likelihood-impact map is just guesswork, and that’s as far as it needs to go at this point. You don’t actually know the real impact of even posited failures such as the ones in your big list; that’s something that chaos engineering experiments will help you find evidence for once you’ve decided which ones are worth the effort of making them into experiments.

First, if you haven’t already done so, put each detailed failure on its own card or sticky note. Then create a large grid with the labels “Impact” and “Likelihood” on it, as shown in Figure 2-2.⁶

Figure 2-2. Empty impact and likelihood grid

Now take each of your specific failure cards or sticky notes and collectively, as a team, discuss where on the map it should be placed. If it’s a high-likelihood failure—perhaps one you’ve seen frequently—with a large impact on customer or user experience, then it should go toward the top right corner of the map. If it’s something you hardly ever expect to see—and something that would likely have little impact on the user experience of your system if it did arise—then the bottom left corner would be the right place for it.

Continue placing your failures onto the map until you and your team have something like the grid shown in Figure 2-3.

Figure 2-3. Failure cards arranged on impact and likelihood grid

You now have a picture of what failures are guessed to be most likely, and an estimate of their potential impact on the user experience. You might feel at this point that you have enough to start building a plan for applying chaos to explore some of your potential failure cases—but wait! There is one final step that will make this map even more useful to you in figuring out what failures to work into chaos experiment hypotheses. It’s time to figure out which failures might actually affect what you care about the most.

Adding What You Care About

You have a map full of potentially worrying failures, organized by your own guesses as to their likelihood and potential impact. You need one more bit of information to turn them into a collection you can navigate and pick from when building your Hypothesis Backlog: an estimate of what the value would be to you and your team, and even your organization, if you were to build trust and confidence in how the system reacts to a particular failure.

This might sound like a complicated thing to add, but it’s usually simpler than you think, though it does require thought and discussion. You need to build another list of the things you, your team, and your company care about. Call it a “taxonomy of -ilities.”⁷ Ask “What do we care about?” and you’ll likely come up with a list that includes the following:

• Reliability

• Safety

• Security

• Durability

• Availability

Some of these may require discussion within the team about what everyone thinks the term might mean for your context, but as long as you all agree, then your list is fine regardless of what it contains. It is your list of things you care about.

Now take another look at each of the failures on your map and mark what “-ilities” you think might be affected if that failure happened. Once again, these are guesses, and it’s not wrong to conclude that every failure will affect all of the things you care about. When you’re done, each failure will look like the example shown in Figure 2-6.

Figure 2-6. Failure card with contribution added

You now have a collective understanding of:

• What failures you can guess might happen

• What impact you can guess a particular failure might have

• What likelihood you guess a failure might have of occurring

• What things you care about might be affected if a given failure occurs

This means that you now have everything you need to take the final step: creating your Hypothesis Backlog!

Creating Your Hypothesis Backlog

A chaos experiment starts with a hypothesis, and with everything you’ve already prepared, the next steps should be easy—not just converting a failure into a hypothesis in a backlog, but also prioritizing which hypotheses might be worth taking to the next level by turning them into a full chaos experiment for a Game Day (see Chapter 3) or even a fully automated chaos experiment (see Chapters Chapter 4 and Chapter 5).

Discuss with your team which failures are worth making into hypotheses. You should have all you need to make that decision, based on combinations such as:

• Selecting the high-impact, high-likelihood failures that map most closely to the most important “ilities”

• Selecting the lowest-impact failures so that you can explore chaos engineering as safely as possible

• Selecting the failures that have the most “-ilities,” and therefore perhaps the greatest value, to be explored further

Your map of failures supplies the raw data. It is now up to you and your team to decide what criteria for selection to apply. Eventually you will have a subset of failures to focus on, and converting them into hypotheses simply requires you to shift the terminology of the failure from “what happens and its impact” to “when it happens it will not have the impact.”

For example, take the failure card from Figure 2-6. You can convert that into a hypothesis card by switching the emphasis as demonstrated in Figure 2-7.

Figure 2-7. Hypothesis card

Now your newly minted hypothesis card reads: “The system will meet its SLOs if database cluster (DB1) becomes unavailable,” where SLO stands for Service Level Objective. This hypothesis is worth considering for prioritizing as an experiment because of where the card is placed on the likelihood-impact map (a high-impact failure), and because of its importance in contributing to building trust and confidence in your system in terms of the “-ilities:” durability and availability.