Earlier in this chapter we discussed Ceph settings defined in the ceph.conf config file. When present as each Ceph daemon starts up, these settings override compiled-in defaults. To borrow a term from the network router world, ceph.conf is an example of stored, startup or saved config. As circumstances cause us to adjust or augment these explicit file-based settings we can effect them by rolling restarts of Ceph daemon processes or even server reboots.

While Ceph's resilient and fault-tolerant architecture affords us the flexibility to carefully apply configuration changes by service restarts, there is faster and less disruptive method: injection.

Ceph's injection mechanism is a handy and nearly immediate way to affect Ceph's running config, which is the set of values that each running daemon process holds within its memory address space. When we inject changes, we avoid the time and potential disruption that a sequential rolling restart of hundreds or thousands of Ceph daemons would entail.

Let's explore some real-world examples of the value of injection, and cover two important caveats as well.

One common use-case for injection is to adjust settings that affect the rate and distribution of backfill and recovery operations. Ceph initiates these as required to maintain data replication as OSDs come up and down or are added or removed from the cluster. It is natural to wish this process to complete as quickly as possible so that the cluster enjoys optimal resilience. Recovery operations, however, can contend with a client I/O; during heavy recovery, users may notice significant performance degradation or even failed operations.

When we anticipate heavy recovery traffic, say if we plan to remove an entire OSD node from service, we can use injection to temporarily throttle recovery to ensure that it does not overwhelm routine operations. First, let's consider the following set of Ceph settings and their default values as of Ceph's Hammer release

For many Hammer clusters, these default values are overly aggressive and can precipitate user impact and even dreaded slow requests. We can lower (or increase) them on a temporary or experimental basis to accommodate planned maintenance or to converge on an optimal balance between recovery speed and client impact. Each Ceph OSD daemon maintains its own running state, and we can inject new values into one or all of them with a single command issued from an admin or MON node.

# ceph tell osd.* injectargs '--osd_max_backfills 1 --osd_recovery_max_active 1 --osd_recovery_max_single_start 1 --osd_recovery_op_priority 1'
osd.0: osd_max_backfills = '1' osd_recovery_max_active = '1' (unchangeable) osd_recovery_max_single_start = '1' (unchangeable) osd_recovery_op_priority = '1' (unchangeable)
osd.1: osd_max_backfills = '1' osd_recovery_max_active = '1' (unchangeable) osd_recovery_max_single_start = '1' (unchangeable) osd_recovery_op_priority = '1' (unchangeable)
    ...

Note the (unchangeable) tags appended to the results for three out of our four variables. This is intended to alert us to values that we can inject, but which do not take effect immediately. Not all values change behavior when injected, some may not be injected at all. In the above example, these messages would lead us to believe that our efforts are in vain and that we must change ceph.conf and restart all OSDs to effect the new values. As of Ceph's Jewel release, though, the code that issues these warnings does not always accurately determine the effectiveness of injection for certain values, including those above.

Experience tells us that these particular values do in fact take effect immediately. Until the code is improved, unfortunately, one must either determine the truth via tests or by researching the experiences of others within the community. Other values known to this author to be effective when injected include

• osd_scrub_max_interval
• osd_deep_scrub_interval
• mon_pg_warn_max_per_osd

Alert readers may infer from the third setting above that one can also successfully inject values into not only OSDs but also Ceph MONs.

An important limitation of injection is that since it modifies only the running config and not the stored config, changes are lost when each Ceph daemon or server is restarted or reboots. Configuration via injection is invaluable for effecting immediate change with minimal disruption, especially when testing the efficacy of a given set of values. Injection helps us respond quickly to incidents within our clusters, and also to temporarily optimize behavior profiles during maintenance operations. Settings that we wish to persist permanently must be also be written to the stored config in ceph.conf or they will be lost in the future as servers are restarted.