We will use the following commands for testing user, bucket, and object sync between the master and secondary zones:

1. Let's create an s3 user in the master site and check if it gets synced to the secondary site.  You should run the following commands in the master site RGW node us-east-1:

        # radosgw-admin user create --uid=pratima 
                                    --display-name="Pratima Umrao"
                                    --id rgw.us-east-1

2. Check the number of users in the master site:

        # radosgw-admin metadata list user --id rgw.us-east-1

3. Let's verify that the user "pratima" got synced to the secondary site. You should run the following commands in the secondary site RGW node us-west-1:

        # radosgw-admin metadata list user --id rgw.us-west-1

        # radosgw-admin user info --uid=pratima --id rgw.us-west-1

We will use the s3cmd application in both sites to create the buckets and upload the objects to these buckets and see if the buckets and objects are getting synced on both the sites.

4. Install and configure s3cmd at the master site us-east-1 node:

5. Configure s3cmd at the master site in the us-east-1 RGW node:

6. Update .s3cfg file options host_base and host_bucket with the master site RGW node hostname us-east-1.cephcookbook.com:

7. Install and configure s3cmd at the secondary site us-west-1 node:

8. Configure s3cmd at the secondary site in the us-west-1 RGW node:

9. Update .s3cfg file options host_base and host_bucket with the master site RGW node hostname us-west-1.

10. Create the test-bucket-master and upload: '/etc/hosts' object in the bucket test-bucket-master:

11. You can now check in the secondary site that the bucket and objects are synced: 

12. Now you can do the opposite to check the active-active replication; for this, you need to create the test-bucket-secondary and upload: '/root/anaconda-ks.cfg' objects in the bucket  test-bucket-secondary at the secondary RGW node us-west-1:

13. You can now check in the master site that the bucket test-bucket-secondary and anaconda-ks.cfg objects are synced:

You can see that the bucket test-bucket-secondary and anaconda-ks.cfg objects are synced to the master site us-east-1. This is the behavior of active-active replication.