Swift, as we already know, is the object store service that stores BLOBs and their metadata. The important part is that multiple copies of the object are stored for redundancy and resiliency. In most organizations, three copies of data are considered good from the redundancy standpoint, but we can choose to have more or fewer copies.
Naturally, more than one node should be used to store the objects. For the purpose of this book, we will just use one storage node—that we have also used for Cinder in the previous section—but this will have a third drive added to it, and use that for Swift.
Swift is a distributed system. In order to store and find data and ensure its integrity and redundancy, the architecture revolves around the concept of rings. Rings essentially are configuration-cum-database files that help in placing the object on the nodes and searching through them.
The rings essentially map the data to the physical device where the data is being stored. There are three rings:
So in effect, if we are looking for all the objects stored for a particular account, all the three rings are checked.
The preceding diagram shows the different components of the Swift service, where ring is the key component. Rings are actually modified versions of the consistent hashing rings.
Let us take an example of the object ring. Say we have three nodes. While creating, we partition the ring in to multiple parts. Since we are using the modified version of the consistent hashing ring, we will get an equal partition size. The partitions are essentially blocks from different nodes where the objects are stored depending on the name of the file.
We create the partition size at the beginning of the ring creation, and it cannot be changed once created. The larger the partition size, the bigger the ring will be and the smoother the object distribution amongst its nodes. Once the partition size is created, 2^Part_Number is the number of partitions created in the ring.
Tip
Rule of thumb states that we need 100 partitions per physical disk that is being used for Swift, and hence, the part power is calculated by the following formula:
log2(number of disks * 100)
Always choose the number on the larger side, as these days the memory (a few megabytes of it) is not a big deal.
Note that the ring partition is not to be confused with the disk partition.
The Swift proxy is the frontend between the object store nodes and the users. In our installation, we will install the proxy on the OSControllerNode and the ObjectStore on the OSStorageNode.
We will install the Swift proxy and some helper packages on the controller node. Let us prepare the checklist so we have all the information handy:
|
Name |
Info |
|---|---|
|
Access to the Internet |
Yes |
|
Proxy Needed |
No |
|
Proxy IP and port |
Not Applicable |
|
Node name |
|
|
Node IP address |
172.22.6.95 |
|
Node OS |
Ubuntu 14.04.1 LTS |
|
Swift Keystone password |
|
|
Swift port |
|
Unlike other services, Swift doesn't need access to the database. Hence, we won't create one. This is because all the data Swift needs is kept in the rings.
We will install the Swift proxy packages and fall back on our reliable aptitude package manager for this:
apt-get install swift swift-proxy python-swiftclient python-keystonemiddleware memcached
Ensure these packages are installed. Now, we see some additional packages in this list. Let us talk about them for a moment. The memcached package caches the objects to serve them fast to the end users, Swift client to configure the system, and the Python modules for the Keystone and Keystone middleware.
We also need to understand the difference in the terminologies when it comes to authentication between Swift and Keystone. These have been explained as follows:
|
Swift |
Keystone |
|---|---|
|
Account |
Tenant / Project |
|
User |
User |
|
Group |
Role |
In order to map the two preceding authentication definitions, the Keystone middleware is used. Another point to be noted is that the Swift user doesn't have any rights by default, but there is a user called the swift operator, which can modify the ACLs on the files. This mapping is also done by the Keystone middleware.
Once the packages are installed, we can move on to the configuration.
The configuration steps are similar to the other services:
- Create a Keystone user and map the roles
- Create a service in Keystone
- Create an endpoint
- Modify configuration files
We start by exporting the credentials. Since we have saved it in the file, we will just source the file by typing source ~alokas/os.txt:
keystone user-create --name swift --pass sw1ftkeypwd
Once the user is created, we will then make it an admin user:
keystone user-role-add --user swift --tenant service --role admin
We create the service by using the following command:
keystone service-create --name swift --type object-store --description "OpenStack Object Storage"
Note down the ID that we will use in the next step of creating the endpoint (in this case, it is febc806b960b496bb3e000fefe992e2b).
We will create the Swift endpoint with the following command:
keystone endpoint-create --service-id febc806b960b496bb3e000fefe992e2b --publicurl 'http://oscontrollernode:8080/v1/AUTH_%(tenant_id)s' --internalurl 'http://oscontrollernode:8080/v1/AUTH_%(tenant_id)s' --adminurl http://oscontrollernode:8080 --region dataCenterOne
The Swift packages don't come with the configuration files. So, we will need to download some sample configuration files from GitHub (https://raw.githubusercontent.com) and then modify them:
mkdir /etc/swift chown -R swift:swift /etc/swift
This creates a directory for Swift configuration files.
We will first download the swift.conf file from the GitHub repository. The following command downloads the sample configuration in the directory that we just created:
curl -o /etc/swift/swift.conf https://raw.githubusercontent.com/openstack/swift/master/etc/swift.conf-sample
In this file, we will have to choose a unique suffix and prefix for our environment. Remember that once chosen, the prefix cannot be changed. In this case, we will use packtpub for our prefix and suffix. This prefix and suffix are used in the hashing algorithm.
Edit the /etc/swift/swift.conf file as follows:
- In the
[swift-hash]section:swift_hash_path_suffix = packtpubsuffix swift_hash_path_prefix = packtpub
There will already be default be a storage policy 0 in the file, verify the presence of the following:
[storage-policy:0] name = Policy-0 default = yes
As a next step, we will download the proxy-server configuration file and modify its configuration:
/etc/swift/proxy-server.conf
https://raw.githubusercontent.com/openstack/swift/master/etc/proxy-server.conf-sample
We will make the following changes to the file:
- Under the
[DEFAULT]section of the configuration, we will mention the user account it would use, the configuration directory, and the port on which it would bind—we have chosen8080:bind_port = 8080 swift_dir = /etc/swift user = swift
- In the
[pipeline:main]section we will enable the modules:pipeline = authtoken cache healthcheck keystoneauth proxy-logging proxy-serverThis allows for the logging and Keystone authentication.
- In the
[app:proxy-server]section, we will enable account management:allow_account_management = true account_autocreate = true
- In the
[filter:authtoken]section, we will configure the Keystone details. Thedelay_auth_decisionvalue is set totrueso that Swift waits until the Keystone middleware and Keystone check the user token and respond to Swift:paste.filter_factory = keystonemiddleware.auth_token:filter_factory auth_uri = http://OSControllerNode:5000/v2.0 identity_uri = http://OSControllerNode:35357 admin_tenant_name = service admin_user = swift admin_password = sw1ftkeypwd delay_auth_decision = true
- In the
[filter:keystoneauth]section, we configure the operator role, which is effectively a mapping that mentions whichroleof Keystone will be considered an operator in Swift. These should exist in the configuration; we can just uncomment the lines rather than having to retype them:use = egg:swift#keystoneauth operator_roles = admin,_member_
- Finally, in the
[filter:cache]section, we configure the memcached location, which is the current node in our case:memcache_servers = 127.0.0.1:11211
The file should appear as seen in the following screenshot:
This concludes the installation of the Swift configuration on the controller node. Please note that in our case, the controller is also the Swift proxy server. In a production environment, we will have to perform the steps on all the different nodes acting as the proxy as the proxy server.
Since we are using the same storage node that we used for Cinder, we already have the DNS/Hosts file figured out. If we choose to have more than one storage node, the same principles apply. We will quickly create a single node.
The storage nodes use rsync in order to keep multiple copies of data in sync. Also, the XFS filesystem works very well for the BLOB storage, so we will install both of those packages:
apt-get install xfsprogs rsync
We will use the fdisk and create two partitions in it using the third drive that we have mounted (/dev/sdc). Alternatively, we could choose to partition from an already existing drive. Let us make it into an XFS filesystem and then mount it to a directory. The reason we have created two partitions is to distribute data uniformly. In the production environment, there will be several nodes and several drives per node, and we can choose to create just one partition per drive.
Check that the drive for Swift is visible to the system by looking for the /dev/sdc in the output of fdisk –l. We will format and partition the disk using the following command:
fdisk /dev/sdc
We will choose the option n to create a new partition, then p for primary, and then we will choose the partition number 1. We will leave the initial sector as default, and for the final sector, we will set the partition size as 50 percent of the disk size. In my case, since the disk is 100 GB, I will create the first partition as 50 GB and hence use +50GB for the last sector.
We will repeat the process and leave everything to default for the second partition, and it will use the remaining space, which is the remaining 50 GB. We will then write to the partition table and come out of the fdisk utility, and at the end of this, we will end up with two partitions: /dev/sdc1 and /dev/sdc2.
You can see the screenshot for the fdisk utility earlier in this chapter, when we created the partition for the Cinder volumes.
As a next step, we will create filesystems on these partitions. XFS is especially suited for an object store; hence, we will use it:
mkfs.xfs /dev/sdc1 mkfs.xfs /dev/sdc2
We will now create folders and mount them:
mkdir -p /srv/node/sdc1 mkdir -p /srv/node/sdc2
Then, we add the new mount points in the /etc/fstab file:
echo "/dev/sdc1 /srv/node/sdc1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 2 " >> /etc/fstab echo "/dev/sdc2 /srv/node/sdc2 xfs noatime,nodiratime,nobarrier,logbufs=8 0 2 " >> /etc/fstab
We will mount the drives using mount –a command and df –k command to verify.
Since we only have one node, we don't actually need to configure rsync. However, it's a good practice to do so because it makes it easier to add nodes in the future.
Add the following to /etc/rsyncd.conf (after replacing the IP address of the storage node):
uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 172.22.6.96 [account] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/account.lock [container] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/container.lock [object] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/object.lock
In the /etc/default/rsync file, set RSYNC_ENABLE to true and start the service using service rsync start.
We will install the account, container, and object components:
apt-get install swift swift-account swift-container swift-object chown swift:swift /etc/swift
Ensure that the installation is successful.
We will also change the permissions of the /srv/node folder:
chown -R swift:swift /srv/node
There are three configuration files that we need to modify; we will download a sample copy from the repository:
curl -o /etc/swift/account-server.conf https://raw.githubusercontent.com/openstack/swift/stable/juno/etc/account-server.conf-sample curl -o /etc/swift/container-server.conf https://raw.githubusercontent.com/openstack/swift/stable/juno/etc/container-server.conf-sample curl -o /etc/swift/object-server.conf https://raw.githubusercontent.com/openstack/swift/stable/juno/etc/object-server.conf-sample
Once the files are downloaded, make the following changes in the three different configuration files.
In the /etc/swift/account-server.conf file, we need to ensure that the following settings are present, and are needed:
- Under the
[Default]section:bind_ip = 172.22.6.96 bind_port = 6002 user = swift swift_dir = /etc/swift devices = /srv/node
- Under the
[pipeline:main]section, we will enable the account server:pipeline = healthcheck recon account-server - Under the
[filter:recon]section, just set up the metrics path:recon_cache_path = /var/cache/swift
In the /etc/swift/container-server.conf file, make the same changes as in case of the account server, but replace the bind_port to 6001, and under the pipeline:main section, replace account-server with container-server, if it already doesn't exist.
In the file /etc/swift/object-server.conf, make the same changes as in case of the account server, but replace the bind_port to 6000, and under the pipeline:main section, replace account-server with object-server, if it doesn't already exist.
We will now create the recon directory and ensure its proper ownership:
mkdir -p /var/cache/swift chown -R swift:swift /var/cache/swift
We will also copy the /etc/swift/swift.conf file from the controller node to here:
scp root@OSControllerNode:/etc/swift/swift.conf /etc/swift/swift.conf
The most important part of the configuration is creating the rings. We will create three rings: one for account, one container, and one object ring. In order to create this, we need to select some values, which we will put in the checklist presented in this section.
We remember from the initial information of Swift that the Swift partition on the ring is technically just directories, and we discussed the rule of thumb formula to choose the partition size.
Since we have two disk partitions (/dev/sdc1 and sdc2) that are being used for object storage, we will substitute in the formula and get log 2 (2 *100). If you are using more than one node, the total numbers of disks need to be taken from all the nodes.
So, our log base 2 calculation yields 7.64, and so rounding it off to the next whole number, we set the partition size to 8. Since our data is not very important, we can live with two copies of it. In a production environment, we will use at least three copies. As discussed in the following table:
|
Name |
Info |
|---|---|
|
Part size |
8 |
|
No of replicas needed |
2 |
|
Minimum time between moving a partition |
1 hour |
|
No. of regions |
1 |
|
No. of zones |
1 |
If you are familiar with AWS, the regions and zones shown in the table are similar to the concept of regions and zones in AWS.
All the rings are created by a utility called swift-ring-builder.
We create the account.builder file with three arguments, namely, partition size, number of replicas and minimum time:
cd /etc/swift swift-ring-builder account.builder create 8 2 1
Once we have created the ring, we will have to add the nodes in there using the command format:
swift-ring-builder account.builder add r1z1-management_ip_of_storage_node:6002/device_name weight
We choose port 6002 for the account, 6001 for container, and 6000 for object in our configuration files above, and we have disk partitions sdc1 and sdc2. The weightage is a relative number when compared to other nodes; it is recommended that we keep it directly proportional to the amount of storage on the drive.
swift-ring-builder account.builder add r1z1-172.22.6.96:6002/sdc1 100 swift-ring-builder account.builder add r1z1-172.22.6.96:6002/sdc2 200
The preceding commands create the ring; we will follow the exact same steps for the container and object rings.
We create the ring with the same parameters as in the preceding section:
swift-ring-builder container.builder create 8 2 1
We then add the drives:
swift-ring-builder container.builder add r1z1-172.22.6.96:6001/sdc1 100 swift-ring-builder container.builder add r1z1-172.22.6.96:6001/sdc2 200
We create the ring with the same parameters:
swift-ring-builder object.builder create 8 2 1
We then add the drives:
swift-ring-builder object.builder add r1z1-172.22.6.96:6000/sdc1 100 swift-ring-builder object.builder add r1z1-172.22.6.96:6000/sdc2 200
This creates the object rings. Once all the rings are created, we rebalance them:
swift-ring-builder container.builder rebalance swift-ring-builder object.builder rebalance swift-ring-builder account.builder rebalance
Now we have three files: account.ring.gz, container.ring.gz, and object.ring.gz in the /etc/swift directory. We need to copy these files to all the other servers running the Swift proxy or the Swift storage components. Since we have created it on the storage node, we will copy it over to the OScontrollerNode:
scp object.ring.gz root@OSControllerNode:/etc/swift scp container.ring.gz root@OSControllerNode:/etc/swift scp account.ring.gz root@OSControllerNode:/etc/swift
As a final step, we will restart the services on the nodes. On the controller node (and where ever else the proxy is installed):
sudo service memcached restart sudo service swift-proxy restart
On all the object store nodes:
swift-init all start
This should start all the storage components. If there are errors in the configuration files, the services will show them here.
In order to validate, just execute the swift stat command, and you will get the output stating that Swift is configured:
We should now be able to create objects and upload them. But we will park the thought for now and revisit this when we test our cloud.