Adding Logging to Your Chaos Experiments

Centralized logging, made up of meaningful logging events, is a strong foundation of any system observability strategy. Here you’re going to add your automated chaos experiment execution into your centralized set of logging events, adding events for all the different stages an experiment goes through when it is run.

You will implement a logging control that listens to a running chaos experiment and send events to a centralized logging system. You are going to see log events being pushed to the Humio centralized logging system, as that is one of the existing implementations available in the Chaos Toolkit Incubator.

The following code sample, taken from a full logging control, shows how you can implement a Chaos Toolkit control function to hook into the life cycle of a running chaos experiment:

...

def before_experiment_control(context: Experiment, secrets:
                              Secrets):
# Send the experiment

    if not with_logging.enabled:
        return

    event = {
        "name": "before-experiment",
        "context": context,
    }
    push_to_humio(event=event, secrets=secrets)

...

With the Humio extension installed, you can now add a controls configuration block (see “Enabling Controls”) to each experiment that, when you execute it, will send logging events to your logging system:

{
    ...

    "secrets": {
        "humio": {
            "token": {
                "type": "env",
                "key": "HUMIO_INGEST_TOKEN"
            },
            "dataspace": {
                "type": "env",
                "key": "HUMIO_DATASPACE"
            }
        }
    },
    "controls": [
        {
            "name": "humio-logger",
            "provider": {
                "type": "python",
                "module": "chaoshumio.control",
                "secrets": ["humio"]
            }
        }
    ]
    ...
}

Alternatively, you can add a global entry into your ~/.chaostoolkit/settings.yaml file to enable the Humio control:

controls:
    humio-logger:
        provider:
            type: python
            module: chaosstuff.control
            secrets: humio

Centralized Chaos Logging in Action

Once that is configured, and the logging extension is installed, you will see logging events from your experiment arriving in your Humio dashboard, as shown in Figure 10-1.

Figure 10-1. Chaos experiment execution log messages

Your chaos experiment executions are now a part of your overall observable system logging. Those events are now ready for manipulation through querying and exploring (see Figure 10-2), like any other logging events.

Figure 10-2. Querying chaos experiment executions

Tracing Your Chaos Experiments

Distributed tracing is critical to comprehending how an interaction with a running system propagates across the system. By enriching your logging messages with trace information, you can piece together the crucial answers to questions such as what events happened, in what order, and who instigated the whole thing. To understand how chaos experiments affect a whole system, you need to add your chaos experiments to the tracing observability picture.

Here you’re going to see how you can use a Chaos Toolkit control that will be able to push trace information into distributed tracing dashboards so that you can view your chaos experiment traces alongside your regular system interaction traces.

Applying the OpenTracing Control

After you have installed the OpenTracing Chaos Toolkit extension, your experiments can be configured to use the OpenTracing control by specifying a configuration block:

{
    "configuration": {
        "tracing_provider": "jaeger",
        "tracing_host": "127.0.0.1",
        "tracing_port": 6831
    },
    "controls": [
        {
            "name": "opentracing",
            "provider": {
                "type": "python",
                "module": "chaostracing.control"
            }
        }
    ]
}

This configuration turns on the control and points the OpenTracing feed at a destination. The destination in this case is a Jaeger tracing visualization dashboard, but it can be any tool that supports receiving an OpenTracing feed.

The preceding configuration tells the Chaos Toolkit to send an experiment execution’s traces to the Jaeger dashboard, where those traces can be displayed alongside all the other traces in your runtime system, as shown in Figure 10-3.

Figure 10-3. Application and chaos traces in the Jaeger dashboard

Summary

In this chapter you used existing Chaos Toolkit controls to add logging and distributed tracing operational concerns to your automated chaos experiments. Both of these controls are passive; they simply listen to the execution of your experiment and send information off to their respective systems. Now it’s time to create your own custom control—one that is not passive. Yours is going to provide real control over the execution of an experiment (refer back to Figure 9-1).