Individual objects up to 5 GB in size can be uploaded to OpenStack Object Storage. However, by splitting the objects into segments, the download size of a single object is virtually unlimited. Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as a single object. By splitting objects into smaller chunks, you also gain efficiency by allowing parallel uploads.
Ensure you are logged in to a Ubuntu host that has access to our OpenStack environment on the 192.168.100.0/24 public network. This host will be used to run client tools against the OpenStack environment created. If you are using the accompanying Vagrant environment, as described in the Preface, you can use the controller
node. It has the python-swiftclient
package that provides the swift
command line client.
If you created this node with Vagrant, you can execute the following command:
vagrant ssh controller
Ensure you have set the following credentials (adjust the path to your certificates and key file to match your environment if not using the Vagrant environment):
export OS_TENANT_NAME=cookbook export OS_USERNAME=admin export OS_PASSWORD=openstack export OS_AUTH_URL=https://192.168.100.200:5000/v2.0/ export OS_NO_CACHE=1 export OS_KEY=/vagrant/cakey.pem export OS_CACERT=/vagrant/ca.pem
Carry out the following steps to upload large objects split into smaller segments:
/tmp
as an example file to upload:dd if=/dev/zero of=/tmp/example-1Gb bs=1M count=1024
-S
option, as follows:swift upload test -S 102400000 /tmp/example-1Gb
You will see output similar to the following screenshot that shows the status of each upload:
OpenStack Object Storage is very good at storing and retrieving large objects. To efficiently do this in our OpenStack Object Storage environment, we have the ability to split large objects into smaller objects with OpenStack Object Storage, maintaining this relationship between the segments and the objects that appear as a single file. This allows us to upload large objects in parallel, rather than streaming a single large file. To achieve this, we use the following syntax:
swift upload container_name -S bytes_to_split large_file
Now, when we list our containers under our account, we have an extra container named test_segments
that holds the actual segmented data fragments for our file. Our test container holds the view that our large object is a single object. Behind the scenes, the metadata within this single object will pull back the individual objects from the test_segments
container to reconstruct the large object. The command as follows:
swift list
When the preceding command is executed, we get the following output:
test test_segments
Now execute the following command:
swift list test
The following output is generated:
tmp/example-1Gb
You can also inspect the segments by listing the test_segments
container with the following command:
swift list test_segments
You will get the following output: