Uploading large objects

Individual objects up to 5 GB in size can be uploaded to OpenStack Object Storage. However, by splitting the objects into segments, the download size of a single object is virtually unlimited. Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as a single object. By splitting objects into smaller chunks, you also gain efficiency by allowing parallel uploads.

Getting ready

Ensure you are logged in to a Ubuntu host that has access to our OpenStack environment on the 192.168.100.0/24 public network. This host will be used to run client tools against the OpenStack environment created. If you are using the accompanying Vagrant environment, as described in the Preface, you can use the controller node. It has the python-swiftclient package that provides the swift command line client.

If you created this node with Vagrant, you can execute the following command:

vagrant ssh controller

Ensure you have set the following credentials (adjust the path to your certificates and key file to match your environment if not using the Vagrant environment):

export OS_TENANT_NAME=cookbook
export OS_USERNAME=admin
export OS_PASSWORD=openstack
export OS_AUTH_URL=https://192.168.100.200:5000/v2.0/
export OS_NO_CACHE=1
export OS_KEY=/vagrant/cakey.pem
export OS_CACERT=/vagrant/ca.pem

How to do it...

Carry out the following steps to upload large objects split into smaller segments:

  1. Create a 1 GB file under /tmp as an example file to upload:
    dd if=/dev/zero of=/tmp/example-1Gb bs=1M count=1024
  2. Rather than uploading this file as a single object, we will utilize segmenting to split this into smaller chunks (in this case, 100-MB segments). To do this, we specify the size of the segments with the -S option, as follows:
    swift upload test -S 102400000 /tmp/example-1Gb

    Note

    Note that the size specified by the -S flag is specified in bytes.

    You will see output similar to the following screenshot that shows the status of each upload:

    How to do it...

How it works...

OpenStack Object Storage is very good at storing and retrieving large objects. To efficiently do this in our OpenStack Object Storage environment, we have the ability to split large objects into smaller objects with OpenStack Object Storage, maintaining this relationship between the segments and the objects that appear as a single file. This allows us to upload large objects in parallel, rather than streaming a single large file. To achieve this, we use the following syntax:

swift upload container_name -S bytes_to_split large_file

Now, when we list our containers under our account, we have an extra container named test_segments that holds the actual segmented data fragments for our file. Our test container holds the view that our large object is a single object. Behind the scenes, the metadata within this single object will pull back the individual objects from the test_segments container to reconstruct the large object. The command as follows:

swift list

When the preceding command is executed, we get the following output:

test
test_segments

Now execute the following command:

swift list test

The following output is generated:

tmp/example-1Gb

You can also inspect the segments by listing the test_segments container with the following command:

swift list test_segments

You will get the following output:

How it works...
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset