To understand this better, let's find out the acting set for a PG from our Ceph cluster:

1. Add a temporary object with name hosts to a pool rbd:

        # rados put -p rbd hosts /etc/hosts

2. Check the PG name for object hosts:

        # ceph osd map rbd hosts

If you observe the output, Placement Group (0.e) has an up set [2,4,3] and an acting set [2,4,3]. So, here osd.2 is the primary OSD, and osd.4 and osd.3 are secondary and tertiary OSDs. The primary OSD is the only OSD that entertains write operations from clients. When it comes to read, by default it also comes from the primary OSD; however, we can change this behavior by setting up read affinity.

The OSD's that are up remains in the up set, as well as the acting set. Once the primary OSD is down, it is first removed from the up set and then from the acting set. The secondary OSD is then promoted to become the primary OSD. Ceph recovers PGs of the failed OSD to a new OSD and then adds it to the up and acting sets to ensure high availability. In a Ceph cluster, an OSD can be the primary OSD for some PGs, while at the same time, it can be the secondary or tertiary OSD for other PGs.