To get started with your first bucket, simply select the Create Bucket option from the S3 dashboard. Provide a suitable name for your new bucket. Remember, your bucket name will have to be unique and will have to start with a lowercase character. Next, select a particular Region where you would like your bucket to be created. Although buckets are global entities in AWS, you still need to provide it with a Region option. This comes in handy, especially when you wish to create a bucket close to your location to optimize latency or meet certain regulatory compliances. Also keep in mind that you are not allowed to change the bucket's name after it has been created, so make sure you provide it a correct and meaningful name before you proceed. In this case, I opted to create my bucket in the Oregon region, as shown:

You can optionally enable logging for your bucket as well, by selecting the Set Up Logging option. This will store detailed access logs of your bucket to a different bucket of your choice. By default, logging of a bucket is disabled; however, you can always re-enable it even after your bucket is created. AWS will not charge you for any of the logging that it will perform; however, it will still charge you for the storage capacity that your logs will consume on S3.

Once your details are in place, select the Create option to create your new bucket. The bucket is created within a few seconds, and you should see the following landing page for your bucket as shown here:

You can even create one or more folders in your bucket by selecting the Create Folder option. Folders are just a nice way to represent and categorize your objects more effectively. You can even perform additional operations on your bucket using this dashboard, such as assign permissions, enable logging, versioning, cross-region replication, and so on. We shall be exploring each of these operations in detail throughout this chapter, but for now, let's upload some objects to our newly created bucket.

With your bucket now created, you can easily upload any object to it. But first, let's take a closer look at what an S3 object actually comprises:

With this basic understanding in mind, we are now ready to upload our first object into S3. You can upload objects directly into your buckets or within sub folders that you may have created. To get started, simply select the Upload option from your bucket or folder. In the Upload–Select Files and Folders dialog box, shown in the following screenshot, you can browse and select any video, document, media file, and so on, of your choice and upload it to S3. The wizard also provides you with an advanced enhanced uploader that is basically a Java applet that can help you upload entire folders into S3 with ease.

The enhanced uploader will be installed on your local machine's browser only once per console session. In my case, I opted to upload a single video file to my S3 bucket, and hence opted to select the Add Files option rather than Enhanced Uploader.

Once your required files are loaded into the wizard, start the upload process by selecting the Start Upload button. You can view the transfer process of your individual files by selecting the transfer panel shown here:

Keep in mind though that you can upload files up to 5 TB in size at any given point in time using the Upload—Select the Files and Folders dialog box. Once the file is uploaded to S3, you are ready to view it!

Each uploaded object in S3 is provided with a URL that you can use to view your object using a browser of your choice. The URL is in the following format: https://s3.amazonaws.com/<BUCKET_NAME>/<OBJECT_NAME>. You can view the URL of your object by simply selecting your object from the dashboard and the Properties option, as shown in the following screenshot. Copy the URL presented against the Link attribute and paste it into a web browser of your choice:

But wait! That's really not going to help you much. In fact if you did try and access your object's URL from a browser, you make have ended up with an XML-based access denied warning! Well, that's because all buckets and objects in S3 are set to private by default. You can change this default behavior by simply selecting the Make Public option from the Actions tab. This will modify your object's permissions and enable everyone to view as well as download your object. You can even perform the same action by selecting the Permissions option from the object's Properties tab. Once the object is made public, you can view it using the URL copied earlier.

Now here's the fun part! S3 provides a wide variety of CLI tools using which you can manipulate your buckets and objects; one of the popular ones being S3CMD. In this section, we will walk through some simple steps to install S3CMD on a local Linux box and then check out some cool commands using which you can work with S3.

So what is S3CMD? In simple words it's a Python-based open source tool used to query any cloud storage service that supports and uses the S3 protocol, including Amazon S3 and even Google's cloud storage. S3CMD can be installed and configured on any major Linux OS, such as CentOS, RHEL, Ubuntu, and so on, and even comes with Windows OS support in the form of a commercial tool called S3Express. The main reason I'm talking about S3CMD here is because of its high versatility and use. If you are capable of writing bash scripts and cron jobs, then you can easily perform automated backups of your files and folders in S3 using S3CMD in a few easy steps.

First off, let's get started by installing S3CMD on our trusty Linux box. To do so, simply type in the following command. However, S3CMD requires Python Version 2.6 and above, so make sure this prerequisite is met before you proceed further:

#  wget http://sourceforge.net/projects/s3tools/files/s3cmd/1.6.0/s3cmd-1.6.0.tar.gz

Here as a screenshot of the preceding command:

Once the tar is downloaded, extract its contents using the following command:

# tar -xvfs3cmd-1.6.0.tar.gz

Next, install S3CMD on your Linux box by executing the following command:

# cd s3cmd-1.6.0
# python setup.py install

With this, S3CMD is now successfully installed. The next step is to configure it to work with your Amazon S3. To do so, type in the following command and follow the on screen instructions provided:

# s3cmd --configure

The configuration utility will request an AWSaccess and Secret Key. It will also prompt you to set your operating region, which is set to US by default. You can even enable S3CMD to communicate with Amazon S3 using the https protocol; however, do note that this setting can have a slight impact on S3CMD's overall performance. The entire configuration is saved locally in the Linux user's home directory in a file called .s3cfg. Once done, you are now ready to test your S3CMD! Here are some basic commands that you can use to query your Amazon S3. First up, let's list our Buckets using the following S3CMD command:

# s3cmdlss3://<BUCKET_NAME>

You should see the contents of your bucket listed using this command. If you do not specify the bucket name, then by default the s3cmdls command will print out all the buckets present in that particular region. Next, let's try to upload some data to our bucket. Uploading is performed using the s3cmd put command and conversely, downloads are performed using the s3cmd get command. Type in the following command in your Linux terminal:

# s3cmd put -r /opt s3://<BUCKET_NAME>

This command will recursively upload all the contents present in the /opt directory to the bucket name that you specify. Remember one important thing though! Trailing slashes after the /opt directory would have copied only the directory's content over to the bucket, but not the directory itself.

On a similar note, you can perform a wide variety of operations using the s3cmd tool. For example, you can upload your instance log files to an S3 bucket on a periodic basis. Here is a base snippet of the /etc/logrotate.d/httpd file where we use the s3cmd command with the sync attribute. The sync attribute is a really useful tool for transferring files over to Amazon S3. sync performs conditional transfers, which means that only files that don't exist at the destination in the same version are transferred.

In this snippet, we are assuming that a bucket with the name httpd-logger has already been created in S3. The code will sync the instance's httpd error log file (/etc/httpd/logs/error_log) and the httpd access log file (/etc/httpd/logs/access_log) and transfer them to their respective folders:

BUCKET=httpd-logger
INSTANCE_ID=`curl --silent http://169.254.169.254/latest/meta-data/instance-id`
  /usr/bin/s3cmd -m text/plain sync /etc/httpd/logs/access_log* s3://${BUCKET}/httpd/access_log/instance=${INSTANCE_ID}/
  /usr/bin/s3cmd -m text/plain sync /etc/httpd/logs/error_log* s3://${BUCKET}/httppd/error_log/instance=${INSTANCE_ID}/

Just like we talked about IAM permissions and policies back in Chapter 2, Security and Access Management, security and access management, S3 too provides permissions and policies using which you can control access to both your buckets and the objects they contain. In this section, we will have a quick look at two such methods provided by S3, namely resource-based policies and user-based policies, as follows:

You will notice that a lot of the syntax actually matches up with what we have already seen in Chapter 2, Security and Access Management, while discussing the building blocks of an IAM policy. Well, here, most of the things remain the same. The only notable difference will be the inclusion of the Principal element, which lists the principals or owners that bucket policy controls access for. The Principal element is not mandatory when creating an IAM policy as it is by default the entity to which the IAM policy is going to be attached. The best part of all this is that AWS provides a really easy to use policy generator tool that you can use to interactively set and create your S3 policies with. You can try out and create your own policies at http://awspolicygen.s3.amazonaws.com/policygen.html.

How do you apply bucket policies? Well that's really simple! Select your bucket from the S3 dashboard and from the Properties panel, select the Permissions drop-down menu, as shown in the following screenshot. Here, select the Add bucket policy option. This will bring up a Bucket Policy Editor dialog box using which you can type in your policy or even use the AWS policy generator to create one for yourself interactively.

Remember to save your policy in the Bucket Policy Editor before closing the dialog box.

ACLs are very similar to bucket policies. An ACL is basically a list of grant operations comprising a grantee and a set of permissions granted. Every bucket that you create in S3 along with each object that you upload gets a set of ACLs with them. ACLs are a great way to control which users get access to your buckets and resources, whether they are AWS users or even some random normal user. To view your bucket's or object's ACLs, simply select the bucket or object from the S3 dashboard and select the Properties option. There, select the Permissions drop-down to view the associated ACLs, as shown here:

ACLs have a predefined set of user groups created using which you can configure access control for your buckets. These user groups include Everyone, Any Authenticated AWSUser, Log Delivery, along with your bucket's creator. ACLs can enforce the following set of permissions (read/write) over an object as well as bucket. On the basis of these permissions, a user can perform operations such as uploading new files or delete existing objects. Here's a quick look at the ACL permissions provided by S3 and how they are associated with both buckets and objects:

Now the obvious question running through your mind right now is what do I use for my S3 buckets and objects? ACLs or bucket policies? The answer for this is two folds. First off, understand that the main difference between an ACL and a bucket policy is that an ACL grants access permissions to buckets or objects individually, whereas a bucket policy will help you write a policy that will either grant or deny access to a bucket or its objects. Ideally, you can use ACLs when each object in the bucket needs to be provided with some explicit grant permissions. You will also need to use ACLs instead of bucket policies when your policy's size reaches 20 KB. Bucket policies have a size limit of 20KB, so if you have a very large number of objects and users to grant access to, you might want to consider the use of ACLs.

User-based policies, perhaps the most simple and easy to use, the user-based policies are nothing more than simple IAM policies that you can create and use to manage access to your Amazon S3 resources. Using these policies, you can create users, groups, and roles in your account and attach specific access permissions to them.

Consider this simple example IAM policy in which we grant the user-specific rights to put objects into S3, to get objects, list them, as well as delete them. Notice the syntax remains quite the same as we have seen throughout this book. You can create multiple such IAM policies and attach them to your users and groups as described here in the AWSS3 documentation page: http://docs.aws.amazon.com/AmazonS3/latest/dev/example-policies-s3.html.

The following is an example of the IAM policy:

{
"Statement": [
        {
"Effect":"Allow",
"Action": [ 
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListBucket"
            ],
"Resource":"arn:aws:s3:::<BUCKET_NAME>/*"
        }
    ]
}

Yes! Believe it or not, you can actually use Amazon S3 to host your websites, provided that they are static in nature. How does it work? Well, its quite simple and easy, actually! All you need to do is create a bucket and enable the website hosting attribute on it. Once done, you can easily upload the website's index.html page along with the other static web pages and voila! You got yourself a really simple website up and running in a matter of minutes.

Here's a simple example in which I used my previously created bucket as a website host. All you need to do is select your bucket and from the Properties panel select the Static Website Hosting option as shown here:

Here, you will see an Endpoint (http://<BUCKET_NAME>.s3-website-<REGION>.amazonaws.com) provided to you. This is your website's end URL, which you can copy and paste in a web browser to view your website; however, before you do that, don't forget to make your website public! To do so, copy and paste the following bucket policy in your bucket's Policy Editor dialog box:

{
"Version":"2012-10-17",
"Statement":[
{
	"Sid":"BucketWebsiteHostingPolicy",
"Effect":"Allow",
	"Principal": "*",
"Action":["s3:GetObject"],
"Resource":["arn:aws:s3:::<BUCKET_NAME>/*"
      ]
    }
  ]
}

Save the policy in the bucket policy editor, and then upload your index.html file as well as an optional error.html file to your bucket. Type your endpoint URL in a web browser of your choice, and you should see your website's landing (index.html) page.

Using static website hosting, you can also redirect all requests to an alternate DNS hostname or even to an alternate bucket. Simply select the Redirect all requests to another host name option and provide an alternative bucket's name in the Redirect all requests to field, as shown in the following screenshot. Alternatively, you can even setup Amazon Route53 to act as your DNS provider by providing a few DNS records and a valid new domain name for your website, such as all-about-dogs.com:

Amazon S3 provides an easy-to-use notification feature that can send notifications in case a certain event gets triggered in your bucket. Currently, S3 supports notifications on the following set of events:

To enable the notification service, select your bucket and from the Properties panel, select the Events drop-down menu, as shown. Fill in the required details and click on Save once done to enable the notification service:

The parameters are as follows:

Once all the fields are filled in, click on Save to enable the notification service. Remember to alter the Bucket's access policy to allow S3 to publish statements to SNS using the sample policy snippet provided here. Replace <SNS_TOPIC_ARN> with an actual SNS Topic ARN value (arn:aws:sns:aws-region:account-id:topic-name) and <BUCKET_NAME> with your bucket name:

{
"Version": "2008-10-17",
"Id": "Policy1448937262025",
"Statement": [
  {
"Sid": "Stmt1448937260611",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
   },
"Action": [
"SNS:Publish"
   ],
"Resource": "<SNS_TOPIC_ARN>",
"Condition": {
"ArnLike": {          
"aws:SourceArn": "arn:aws:s3:*:*:<BUCKET_NAME>"
    }
   }
  }
 ]
}

Versioning is perhaps the most important and useful feature provided by S3. In a way, it is a means to create and maintain multiple copies of a single object present in a bucket. Each copy of the same object that you upload into the bucket receives a unique version ID, which can later be used to retrieve and restore the object in case of an accidental deletion or failure. To enable versioning on a bucket, all you need to do is select the particular bucket from the S3 dashboard, and from the Properties panel, select the Versioning drop-down menu, as shown here:

By default, versioning is disabled on all buckets, so you will have to explicitly enable versioning on the ones that require it. Once enabled, all newly uploaded objects will receive a unique version ID. Older objects stored in the bucket prior to enabling versioning also contain a version ID parameter, but the value of that is set to null. However, once versioning is enabled on the bucket, it applies to all the objects contained in the bucket, so all objects uploaded henceforth, new or old, will obtain version IDs. An important point to remember here is that once you have enabled versioning on a bucket, there is no way you can disable it. You can, however, suspend it by selecting the Suspend Versioning option.

So how do you upload versions of your objects to a bucket? Well, the process remains absolutely the same for any object that you upload into S3. The only difference now is that with each new version of the object that you upload, S3 will assign it a unique version ID and store it in the same bucket as its originator. To list the different versions of an object, all you need to do is toggle between the Hide and Show versions buttons as shown in the following screenshot.

Selecting the Show versions will display all the uploaded versions of that object including the object's creation date, version ID, and size. You can then download any of the versioned objects that you want by simply selecting it from the dashboard and, from the Actions drop-down menu, select the Download option. In case the download request is issued on the main object, then S3 will simply fetch the latest uploaded version of that object and download it for you:

Versioning also comes in real handy when you want to protect your data from any accidental deletions or overwrites. How? Well, to put in simple words, when you try to DELETE an object that is versioned in S3, you actually don't wipe it from existence immediately. In fact, S3 will insert something called as a delete marker in the bucket and that marker becomes the current version of the object with a new version ID. When you try to GET an object whose current version is a delete marker, Amazon S3 behaves as though the object has been deleted and returns a 404 error even though that object is not physically deleted from the bucket. To permanently wipe out the object, you will need to use the DELETE object along with its version ID. Want to try it out? Then go ahead and delete a main object from your versioned S3 bucket. You will notice that although the main object is successfully deleted and not visible from the S3 dashboard, its versions are still pretty much intact. So even if this was an accidental DELETE operation, you can still retrieve the main image from the version ID! Amazing, isn't it!

Another sweet part of enabling versioning on a bucket is that you can specify an object's transition or lifecycle as well. This feature comes in real handy when you want the objects stored in your bucket to get auto-archived to, let's say, Amazon Glacier after a long period of storage in S3, or wish to transition the storage class of an object from Standard to Standard_IA for infrequent access. Logs are a classic example of where this feature comes in really handy. Your bucket can store the logs for up to a week's duration using standard storage and then post that you transition the logs to either Glacier for archiving or even delete them permanently. To enable lifecycle management, you will need to select your bucket from the S3 Dashboard and from the Properties panel, select the Lifecycle drop-down menu, as shown:

Next, select the Add rule option to create a lifecycle rule for your bucket. This will pop up the Lifecycle Rules wizard using which you can configure as well as choose the target objects for which you wish to enable the lifecycle management. The first page of the wizard is Choose Rule Target, where you can specify either a prefix or the entire bucket to which the lifecycle rules shall apply. A prefix is more or less like a folder name that you can specify, for example, in my case I provided the prefix Videos/, which is an actual directory inside my bucket. Select Configure Rule to move on to the next page of the wizard.

In the Action on Current Version page, you get to choose and specify the type of action you wish to perform over your selected objects. For example, you can enable an object's transition to the Standard - Infrequent Access Storage class 30 days after the object's creation date, which is the ideal time to set for an object, or you can even enable Archiving to the Glacier Storage Class post 60 days your object's creation, and so on. There are a few rules that you have to keep in mind, however, when performing transition actions:

The following screenshot shows the lifecycle configuration:

If you are happy with your lifecycle configuration, click on Review to complete the process. In the Review and Name pages, you can specify an optional name for this particular transition rule as well. Make sure you review your rules correctly before selecting the Create and Activate Rule options. Post your rules creation. You can further edit or even delete them from the same Lifecycle drop-down menu found in the Properties panel of the S3 dashboard. To know more about lifecycle management and how you can leverage it for your objects and buckets, check http://docs.aws.amazon.com/AmazonS3/latest/UG/LifecycleConfiguration.html.

Versioning in S3 also provides us with yet another easy to use feature, which is called Cross-Region Replication. As the name suggests, this feature enables you to copy the contents of your bucket or the bucket itself asynchronously to a bucket present in some different AWS region. The copy process will basically copy all aspects of your source bucket and objects, including their creation date and time, version IDs, metadata, and so on, over to the destination bucket.

To enable Cross-Region Replication on a bucket, all you need to do is first make sure that the bucket has versioning enabled. Next, from the Properties panel, select the option Cross-Region Replication, as shown here:

Click on Enable Cross-Region Replication to get things started. You can either select the entire bucket as the Source or even specify a prefix in the bucket, for example, Images/. Next, select an appropriate Destination Region for your bucket replication. In my case, I opted for the Northern California region. Select the Destination Bucket of your choice as well. This destination bucket is not auto created by S3, so it's your responsibility to go ahead and create one in the destination region of your choice:

You can optionally change the Destination Storage Class of your replicated objects as well. For example, if you wish to minimize your costs of replication, you can instruct S3 to store all replicated objects in the destination bucket using the Standard-IA storage class. Finally, you will also need to setup an IAM Role to grant S3 permission to replicate objects on your behalf. Click on Create/ Select IAM Role to bring up the IAM Dashboard. There, select the Roles option from the navigation pane and attach the following role policy:

{
"Version":"2012-10-17",
"Statement":[
      {
"Effect":"Allow",
"Principal":{
"Service":"s3.amazonaws.com"
         },
"Action":"sts:AssumeRole"
      }
   ]
}

Once done, S3 will asynchronously copy the contents from your source bucket over to the destination bucket. This includes any new objects that you add to the source bucket as well. An important point to note here is that using Cross-Region Replication, S3 will only replicate your actions over to the destination bucket, such as adding, copying, and deleting objects. Lifecycle configuration actions, such as transitioning objects from Standard to Standard-IA or to Glacier are not replicated. You can however, configure the same lifecycle configurations as your source bucket over to your destination buckets manually.