If you want to add the new OSD node in production peak hours or non-peak hours and you want to have the least impact in client IO as compared to Ceph data rebalance - recovery and backfill IO due to new OSD new. You can throttle the backfill and recovery with the help of following commands:
- Set osd_max_backfills = 1 option to throttle the backfill threads. You can add this in ceph.conf [osd] section and you can also set it dynamically with the following command:
# ceph tell osd.* injectargs '--osd_max_backfills 1'
- Set osd_recovery_max_active = 1 option to throttle the recovery threads. You can add this in ceph.conf [osd] section and you can also set it dynamically with the following command:
# ceph tell osd.* injectargs '--osd_recovery_max_active 1'
- Set osd_recovery_op_priority = 1 option to lower the recovery priority. You can add this in ceph.conf [osd] section and you can also set it dynamically with the following command:
# ceph tell osd.* injectargs '--osd_recovery_op_priority 1'
With the Jewel release of Ceph, there are two additional flags that are enabled by default when a Ceph cluster is installed at Jewel. If the cluster was upgraded from a version prior to Jewel (Hammer for example) these flags can be enabled:
- sortbitwise: The sortbitwise flag indicates that objects are sorted in a bitwise fashion. The old sort order nibblewise, was an historical artifact of filestore that is simply inefficient with the current version of Ceph. Bitwise sort order makes operations that require listing objects, like backfill and scrubbing, a bit more efficient:
# ceph osd set sortbitwise
- require_jewel_osds: This flag prevents any pre-Jewel OSDs from joining the Ceph cluster. The purpose of this flag is to prevent an OSD from joining the cluster that will not support features that the Jewel code supports leading to possible OSD flapping and cluster issues:
# ceph osd set require_jewel_osds
In addition to these flags, you can also use the following commands to repair OSDs and PGs:
- ceph osd repair: This performs repairing on a specified OSD.
- ceph pg repair: This performs repairing on a specified PG. Use this command with caution; based on your cluster state, this command can impact user data if not used carefully.
- ceph pg scrub: This performs scrubbing on a specified PG.
- ceph deep-scrub: This performs deep-scrubbing on specified PGs.
The Ceph CLI is quite powerful for end-to-end cluster management. You can get more information at http://docs.ceph.com/docs/master/rados/man/.
d manager service. Each time you start, restart, and stop Ceph daemons (or your entire cluster), you must specify at least one option and one command. You may also specify a daemon type or a daemon instance. The general syntax for this is as follows:
systemctl [options...] command [service name...]
The systemctl options include:
- --help or -h: Prints a short help text
- --all or -a: When listing units, show all loaded units, regardless of their state
- --signal or -s: When used will kill, choose which signal to send to the selected process
- --force or -f: When used with enable, overwrite any existing conflicting symlinks
- --host or -h: Execute an operation on a remote host
The systemctl commands include the following:
- status: Shows status of the daemon
- start: Starts the daemon
- stop: Stops the daemon
- restart: Stops and then starts the daemon
- kill: Kills the specified daemon
- reload: Reloads the config file without interrupting pending operations
- list-units: List known units managed by systemd
- condrestart: Restarts if the service is already running
- enable: Turns the service on for the next boot or other triggering event
- disable: Turns the service off for the next boot or other triggering event
- is-enabled: Used to check whether a service is configured to start or not in the current environment
systemctl can target the following Ceph service types:
-
- ceph-mon
- ceph-osd
- ceph-mds
- ceph-radosgw