To understand images and instances a bit better, we first need to travel a little back in time; don't worry, a couple of years back is quite enough! This was the time when there was a boom in the implementation and utilization of the virtualization technology!
Almost all IT companies today run their workloads off virtualized platforms such as VMware vSphere or Citrix XenServer to even Microsoft's Hyper-V. AWS, too, got into the act but decided to use and modify a more off the shelf, open sourced Xen as its virtualization engine. And like any other virtualization technology, this platform was also used to spin up virtual machines using either some type of configuration files or some predefined templates. In AWS's vocabulary, these virtual machines came to be known as instances and their master templates came to be known as images.
By now you must have realized that instances and images are nothing new! They are just fancy nomenclature that differentiates AWS from the rest of the plain old virtualization technologies, right? Well, no. Apart from just the naming convention, there are a lot more differences to AWS images and instances as compared to your everyday virtual machines and templates. AWS has put in a lot of time and effort from time to time in designing and structuring these images and instances, such that they remain lightweighted, spin up more quickly, and can even be ported easily from one place to another. These factors make a lot of difference when it comes to designing scalable and fault tolerant application environments in the cloud.
We shall be learning a lot about these concepts and terminologies in the coming sections of this, as well as in the next chapter, but for now, let's start off by understanding more about these images!
As discussed earlier, images are nothing more than preconfigured templates that you can use to launch one or more instances from. In AWS, we call these images Amazon Machine Images (AMIs). Each AMI contains an operating system which can range from any modern Linux distro to even Windows Servers, plus some optional software application, such as a web server, or some application server installed on it.
It is important, however, to understand a couple of important things about AMIs. Just like any other template, AMIs are static in nature, which basically means that once they are created, their state remains unchanged. You can spin up or launch multiple instances using a single AMI and then perform any sort of modifications and alterations within the instance itself. There is also no restriction on the size of instances that you can launch based on your AMI. You can select anything from the smallest instance (also called as a micro instance) to the largest ones that are generally meant for high performance computing. Take a look at the following image of EC2 AMI:
Secondly, an AMI can contain certain launch permissions as well. These permissions dictate whether the AMI can be launched by anyone (public) or by someone or some account which I specify (explicit) or I can even keep the AMI all to myself and not allow anyone to launch instances from it but me (implicit). Why have launch permissions? Well, there are cases where some AMIs can contain some form of propriety software or licensed application, which you do not want to share freely among the general public.
In that case, these permissions come in really handy! You can alternatively even create something called as a paid AMI. This feature allows you to share your AMI to the general public, however, with some support costs associated with it.
AMIs can be bought and sold using something called as the AWS Marketplace as well—a one stop shop for all your AMI needs! Here, AMIs are categorized according to their contents and you as an end user can choose and launch instances off any one of them. Categories include software infrastructure, development tools, business and collaboration tools, and much more! These AMIs are mostly created by third parties or commercial companies who wish to either sell or provide their products on the AWS platform.
Click on and browse through the AWS Marketplace using https://aws.amazon.com/marketplace.
AMIs can be broadly classified into two main categories depending on the way they store their root volume or hard drive:
The following image depicts the deployments of both the instance store-backed and EBS-backed AMIs. As you can see, the root and data volumes of the instance store-backed AMI are stored locally on the HOST SERVER itself, whereas the second instance uses EBS volumes to store its root device and data.
The following is a quick differentiator to help you understand some of the key differences between EB-backed and Instance store-backed AMIs:
Amazon Linux AMI is a specially created, lightweight Linux-based image that is supported and maintained by AWS itself. The image is based off a RedHat Enterprise Linux (RHEL) distro, which basically means that you can execute almost any and all RHEL-based commands, such as yum and system-config, on it.
The image also comes pre-packaged with a lot of essential AWS tools and libraries that allow for easy integration of the AMI with other AWS services. All in all, everything from the yum repos to the AMIs security and patching is taken care of by AWS itself!
The Amazon Linux AMI comes at no additional costs. You only have to pay for the running instances that are created from it. You can read more about the Amazon Linux AMI at http://aws.amazon.com/amazon-linux-ami/.
Later on, we will be using this Amazon Linux AMI itself and launching our very first, but not the last, instance into the cloud, so stick around!
So far we have only being talking about images; so now let's shift the attention over to instances! As discussed briefly earlier, instances are nothing but virtual machines or virtual servers that are spawned off from a single image or AMI. Each instance comes with its own set of resources, namely CPU, memory, storage, and network, which are differentiated by something called as instance families or instance types. When you first launch an instance, you need to specify its instance type. This will determine the amount of resources that your instance will obtain throughout its lifecycle.
AWS currently supports five instance types or families, which are briefly explained as follows:
To know more about the various instance types and their use cases, refer to http://aws.amazon.com/ec2/instance-types/.
As of late, AWS EC2 supports close to 38 instance types, each with their own set of pros and cons and use cases. In such times, it actually becomes really difficult for an end user to decide which instance type is right for his/her application. The easiest and most common approach taken is to pick out the closet instance type that matches your application's set of requirements - for example, it would be ideal to install a simple MongoDB database on a memory optimized instance rather than a compute or GPU optimized instance. Not that compute optimized instances are a wrong choice or anything, but it makes more sense to go for memory in such cases rather than just brute CPU. From my perspective, I have always fancied the general purpose set of instances simply because most of my application needs seem to get balanced out correctly with it, but feel free to try out other instance types as well.
Apart from the various instance types, EC2 also provides three convenient instance pricing options to choose from, namely on-demand, reserved, and spot instances. You can use either or all of these pricing options at the same time to suit your application's needs. Let's have a quick look at all three options to get a better understanding of them.
Pretty much the most commonly used instance deployment method, the on-demand instances are created only when you require them, hence the term on-demand. On-demand instances are priced by the hour with no upfront payments or commitments required. This, in essence, is the true pay-as-you-go payment method that we always end up mentioning when talking about clouds. These are standard computational resources that are ready whenever you request them and can be shut down anytime during its tenure.
By default, you can have a max of 20 such on-demand instances launched within a single AWS account at a time. If you wish to have more such instances, then you simply have to raise a support request with AWS using the AWS Management Console's Support tab. A good use case for such instances can be an application running unpredictable workloads, such as a gaming website or social website. In this case, you can leverage the flexibility of on-demand instances accompanied with their low costs to only pay for the compute capacity you need and use and not a dime more!
Consider this simple example: A t2.micro instance costs $0.013 per hour to run in the US East (N. Virginia) region. So, if I was to run this instance for an entire day, I would only have to pay $0.312! Now that's cloud power!
Deploying instances using the on-demand model has but one slight drawback, which is that AWS does not guarantee the deployment of your instance. Why, you ask? Well to put it simply, using on-demand model, you can create and terminate instances on the go without having to make any commitments whatsoever. It is up to AWS to match this dynamic requirement and make sure that adequate capacity is present in its datacenters at all times. However, in very few and rare cases, this does not happen, and that's when AWS will fail to power on your on-demand instance.
In such cases, you are better off by using something called as reserved instances, where AWS actually guarantees your instances with resource capacity reservations and significantly lower costs as compared to the on-demand model. You can choose between three payment options when you purchase reserved instances: all upfront, partial upfront, and no upfront. As the name suggests, you can choose to pay some upfront costs or the full payment itself for reserving your instances for a minimum period of a year and maximum up to three years.
Consider our earlier example of the t2.micro instance costing $0.0013 per hour. The following table summarizes the upfront costs you will need to pay for a period of one year for a single t2.micro instance using the reserved instance pricing model:
Payment method |
Upfront cost |
Monthly cost |
Hourly cost |
Savings over on-demand |
---|---|---|---|---|
No upfront |
$0 |
$6.57 |
$0.009 |
31% |
Partial upfront |
$51 |
$2.19 |
$0.0088 |
32% |
All upfront |
$75 |
$0 |
$0.0086 |
34% |
Reserved instances are the best option when the application loads are steady and consistent. In such cases, where you don't have to worry about unpredictable workloads and spikes, you can reserve a bunch of instances in EC2 and end up saving on additional costs.
Spot instances allow you to bid for unused EC2 compute capacity. These instances were specially created to address a simple problem of excess EC2 capacity in AWS. How does it all work? Well, it's just like any other bidding system. AWS sets the hourly price for a particular spot instance that can change as the demand for the spot instances either grows or shrinks. You as an end user have to place a bid on these spot instances, and when your bid exceeds that of the current spot price, your instances are then made to run! It is important to also note that these instances will stop the moment someone else out bids you, so host your application accordingly. Ideally, applications that are non-critical in nature and do not require large processing times, such as image resizing operations, are ideally run on spot instances.
Let's look at our trusty t2.micro instance example here as well. The on-demand cost for a t2.micro instance is $0.013 per hour; however, I place a bid of $0.0003 per hour to run my application. So, if the current bid cost for the t2.micro instance falls below my bid, then EC2 will spin up the requested t2.micro instances for me until either I choose to terminate them or someone else out bids me on the same—simple, isn't it?
Spot instances compliment the reserved and on-demand instances; hence, ideally, you should use a mixture of spot instances working on-demand or reserved instances just to be sure that your application has some compute capacity on standby in case it needs it.
Okay, so we have seen the basics of images and instances along with various instance types and some interesting instance pricing strategies as well. Now comes the fun part! Actually deploying your very own instance on the cloud!
In this section, we will be using the AWS Management Console and launching our very first t2.micro instance on the AWS cloud. Along the way, we shall also look at some instance lifecycle operations such as start, stop, reboot, and terminate along with steps, using which you can configure your instances as well. So, what are we waiting for? Let's get busy!
To begin with, I have already logged in to my AWS Management Console using the IAM credentials that we created in our previous chapter. If you are still using your root credentials to access your AWS account, then you might want to revisit Chapter 2, Security and Access Management, and get that sorted out! Remember, using root credentials to access your account is a strict no no!
Once you have logged into the AWS Management Console, finding the EC2 option isn't that hard. Select the EC2 option from under the Compute category, as shown in the following screenshot:
This will bring up the EC2 dashboard on your browser. Feel free to have a look around the dashboard and familiarize yourself with it. To the left, you have the Navigation pane that will help you navigate to various sections and services provided by EC2, such as Instances, Images, Network and Security, Load Balancers, and even Auto Scaling. The centre dashboard provides a real-time view of your EC2 resources, which includes important details such as how many instances are currently running in your environment, how many volumes, key pairs, snapshots, or elastic IPs have been created, so on and so forth.
The dashboard also displays the current health of the overall region as well as its subsequent availability zones. In our case, we are operating from the US West (Oregon) region that contains additional AZs called as us-west-2a, us-west-2b, and us-west-2c. These names and values will vary based on your preferred region of operation.
Next up, we launch our very first instance from this same dashboard by selecting the Launch Instance option, as shown in the following screenshot:
On selecting the Launch Instance option, you will be directed to a wizard driven page that will help you create and customize your very first instance. This wizard divides the entire instance creation operation into seven individual stages, each stage having its own set of configurable items. Let's go through these stages one at a time.
Naturally, our first instance has to spawn from an AMI, so that's the first step! Here, AWS provides us with a whole lot of options to choose from, which includes a Quick Start guide, which lists out the most frequently used and popular AMIs, and includes the famous Amazon Linux AMI as well, as shown in the following screenshot:
There are also a host of other operating systems provided here as well which includes Ubuntu, SUSE Linux, Red Hat, and Windows Servers.
Each of these AMIs has a uniquely referenced AMI ID, which looks something like this: ami-e75272d7. We can use this AMI ID to spin up instances using the AWS CLI, something which we will perform in the coming sections of this chapter. They also contain additional information such as whether the root device of the AMI is based on an EBS volume or not, whether the particular AMI is eligible under the Free tier or not, and so on and so forth.
Besides the Quick Start guide, you can also spin up your instances using the AWS Marketplace and the Community AMIs section as well. Both these options contain an exhaustive list of customized AMIs that have been created by either third-party companies or by developers and can be used for a variety of purposes. But for this exercise, we are going to go ahead and select Amazon Linux AMI itself from the Quick Start menu.
With the AMI selected, the next step is to select the particular instance type or size as per your requirements. You can use the Filter by option to group and view instances according to their families and generations as well. In this case, we are going ahead with the general purpose t2.micro instance type, which is covered under the free tier eligibility and will provide us with 1 VCPU and 1 GB of RAM to work with! The following screenshot shows the configurations of the instance:
Ideally, now you can launch your instance right away, but this will not allow you to perform any additional configurations on your instance, which just isn't nice! So, go ahead and click on the Next: Configure instance Details button to move on to the third stage.
Now here it gets a little tricky for first timers. This page will basically allow you to configure a few important aspects about your instance, including its network settings, monitoring, and lots more. Let's have a look at each of these options in detail:
In our case, the VPC has a default network of 172.31.0.0/16, which means we can assign up to 65,536 IP addresses using it.
Once you have selected your values, move on to the fourth stage of the instance deployment process by selecting the Next: Add Storage option.
Using this page, you can add additional EBS volumes to your instances. To add new volumes, simply click on the Add New Volume button. This will provide you with options to provide the size of the new volume along with its mount points. In our case, there is an 8 GB volume already attached to our instance. This is the t2.micro instance's root volume, as shown in the following screenshot:
You can optionally increase the size of the volume and enable add-on features such as Delete on Termination as per your requirement. Once done, proceed to the next stage of the instance deployment process by selecting the Next: Tag instance option.
The tag instances page will allow you to specify tags for your EC2 instance. Tags are nothing more than normal key-value pairs of text that allow you to manage your AWS resources a lot easily. You can start, stop, and terminate a group of instances or any other AWS resources using tags. Each AWS resource can have a maximum of 10 tags assigned to it. For example, in our case, we have provided a tag for our instance as ServerType:WebServer. Here, ServerType is the key and WebServer its corresponding value. You can have other group of instances in your environment tagged as ServerType:DatabaseServer or ServerType:AppServer based on their application. The important thing to keep in mind here is that AWS will not assign a tag to any of your resources automatically. These are optional attributes that you assign to your resources in order to facilitate in easier management:
Once your tags are set, click on the Next: Configure Security Group option to proceed.
Security groups are an essential tool used to safeguard access to your instances from the outside world. Security groups are nothing but a set of firewall rules that allow specific traffic to reach your instance. By default, the security groups allow for all outbound traffic to pass while blocking all inbound traffic. By default, AWS will auto-create a security group for you when you first start using the EC2 service. This security group is called as default and contains only a single rule that allows all inbound traffic on port 22.
In the Configure Security Groups page, you can either choose to Create a new security group or Select an existing security group. Let's go ahead and create one for starters. Select the Create a new security group option and fill out a suitable Security group name and Description. By default, AWS would have already enabled inbound SSH access by enabling port 22:
You can add additional rules to your security group based on your requirements as well. For example, in our instance's case, we want the users to receive all inbound HTTP traffic as well. So, select the Add Rule option to add a firewall rule. This will populate an additional rule line, as shown in the preceding screenshot. Next, from the Type dropdown, select HTTP and leave the rest of the fields to their default values. With our security group created and populated, we can now go ahead with the final step in the instance launch stage.
Yup! Finally, we are here! The last step toward launching your very first instance! Here, you will be provided with a complete summary of your instance's configuration details, including the AMI details, instance type selected, instance details, and so on. If all the details are correct, then simply go ahead and click on the Launch option. Since this is your first instance launch, you will be provided with an additional popup page that will basically help you create a key pair.
A key pair is basically a combination of a public and a private key, which is used to encrypt and decrypt your instance's login info. AWS generates the key pair for you which you need to download and save locally to your workstation. Remember that once a particular key pair is created and associated with an instance, you will need to use that key pair itself to access the instance. You will not be able to download this key pair again; hence, save it in a secure location. Take a look at the following screenshot to get an idea of selecting the key pair:
Select the
Create a new key pair option from the dropdown list and provide a suitable name for your key pair as well. Click on the Download Key Pair option to download the .PEM
file. Once completed, select the Launch Instance option. The instance will take a couple of minutes to get started. Meanwhile, make a note of the new instance's ID (in this case, i-53fc559a) and feel free to view the instance's launch logs as well:
Phew! With this step completed, your instance is now ready for use! Your instance will show up in the EC2 dashboard, as shown in the following screenshot:
The dashboard contains and provides a lot of information about your instance. You can view your instance's ID, instance type, power state, and a whole lot more info from the dashboard. You can also obtain your instance's health information using the Status Checks tab and the Monitoring tab. Additionally, you can perform power operations on your instance such as start, stop, reboot, and terminate using the Actions tab located in the preceding instance table.
Before we proceed to the next section, make a note of your instance's Public DNS and the Public IP. We will be using these values to connect to the instances from our local workstations.
Once your instance has launched successfully, you can connect to it using three different methods that are briefly explained as follows:
*.ppk
).You can download Putty, PuttyGen, and various other SSH and FTP tools from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html.
After creating your private key, follow the next steps to use Putty and PuttyGen:
PEM
file. Remember, this is the same file that we downloaded during stage 7 of the instance launch phase.PuttyGen will probably prompt you with a warning message stating that you are saving this key without a passphrase and would you like to continue.
*.PPK
) at a secure and accessible location. You can now use this PPK file to connect to your instance using Putty.Now comes the fun part! Launch a Putty session from the Start menu. You should see the Putty dialog box as shown in the following screenshot. Here, provide your instance's Public DNS or Public IP in the Host Name (or IP address) field as shown. Also make sure that the Port value is set to 22 and the Connection type is selected as SSH.
ec2-user
) and hit the Enter key. Voila! Your first instance is now ready for use, as shown in the following screenshot. Isn't that awesome!*.PEM
) file over to the Linux server using and SCP tool. In my case, I always use WinSCP to achieve this. It's a simple tool and pretty straightforward to use. Once the key is transferred, run the following command to change the key's permissions:# chmod 400 <Private_Key>.pem
# ssh -I <Private_Key>.pem ec2-user@<EC2_Instance_PublicDNS>
Once your instances are launched, you can configure virtually anything in it, from packages, to users, to some specialized software or application, anything and everything goes!
Let's begin by running some simple commands first. Go ahead and type the following command to check your instance's disk size:
# df –h
Here is the output showing the configuration of the instance:
You should see an 8 GB disk mounted on the root (/
) partition, as shown in the preceding screenshot. Not bad, eh! Let's try something else, like updating the operating system. AWS Linux AMIs are regularly patched and provided with necessary package updates, so it is a good idea to patch them from time to time. Run the following command to update the Amazon Linux OS:
# sudo yum update -y
Why sudo? Well, as discussed earlier, you are not provided with root privileges when you log in to your instance. You can change that by simple changing the current user to root after you login; however, we are going to stick with the ec2-user itself for now.
What else can we do over here? Well, let's go ahead and install some specific software for our instance. Since this instance is going to act as a web server, we will need to install and configure a basic Apache HTTP web server package on it.
Type in the following set of commands that will help you install the Apache HTTP web server on your instance:
# sudo yum install httpd
Once the necessary packages are installed, simply start the Apache HTTP server using the following simple commands:
# sudo service httpd start # sudo chkconfig httpd on
You can see the server running after running the preceding commands, as shown in the following screenshot:
You can verify whether your instance is actually running a web server or not by launching a web browser on your workstation and typing either in the instance's public IP or public DNS. You should see the Amazon Linux AMI test page, as shown in the following screenshot:
There you have it! A fully functional and ready-to-use web server using just a few simple steps! Now wasn't that easy!
So far, we have seen how to launch and manage instances in EC2 using the EC2 dashboard. In this section, we are going to see how to leverage the AWS CLI to launch your instance in the cloud! For this exercise, I'll be using my trusty old CentOS 6.5 machine, which has been configured from Chapter 2, Security and Access Management, to work with the AWS CLI. So, without further ado, let's get busy!
First up, let's create a new key pair for our instance. Note that you can use existing key pairs to connect to new instances; however, we will still go ahead and create a new one for this exercise. Type in the following command in your terminal:
# aws ec2 create-key-pair --key-name <Key_Pair_Name> > --output text > <Key_Pair_Name>.pem
Once the key pair has been created, remember to change its permissions using the following command:
# chmod 400 <Key_Pair_Name>.pem
And you can see the created key:
Once again, you can very well reuse an existing security group from EC2 for your new instances, but we will go ahead and create one here. Type in the following command to create a new security group:
# aws ec2 create-security-group --group-name <SG_Name> > --description "<SG_Description>"
For creating security groups, you are only required to provide a security group name and an optional description field along with it. Make sure that you provide a simple yet meaningful name here:
Once executed, you will be provided with the new security group's ID as the output. Make a note of this ID as it will be required in the next few steps.
With your new security group created, the next thing to do is to add a few firewall rules to it. We will be discussing a lot more on this topic in the next chapter, so to keep things simple, let's add one rule to allow inbound SSH traffic to our instance. Type in the following command to add the new rule:
# aws ec2 authorize-security-group-ingress --group-name <SG_Name> > --protocol tcp --port 22 --cidr 0.0.0.0/0
To add a firewall rule, you will be required to provide the security group's name to which the rule has to be applied. You will also need to provide the protocol, port number, and network CIDR values as per your requirements:
With the key pair and security group created and populated, the final thing to do is to launch your new instance. For this step, you will need a particular AMI ID along with a few other key essentials such as your security group name, the key pair, and the instance launch type, along with the number of instances you actually wish to launch.
Type in the following command to launch your instance:
# aws ec2 run-instances --image-id ami-e7527ed7 > --count 1 --instance-type t2.micro > --security-groups <SG_Name> > --key-name <Key_Pair_Name>
And here is the output of the preceding commands:
The instance will take a good two or three minutes to spin up, so be patient! Make a note of the instance's ID from the output of the ec2 run-instance command. We will be using this instance ID to find out the instance's public IP address using the EC2 describe-instance command as shown:
# aws ec2 describe-instances --instance-ids <Instance_ID>
Make a note of the instance's public DNS or the public IP address. Next, use the key pair created and connect to your instance using any of the methods discussed earlier.