CloudWatch provides a really simple alarm setup using which you as an end user can monitor your account's estimated costs and usage. To work with this, you need to log in to your AWS account as the root user and not as an IAM user, even if you are the administrator. I know I'm not following my own rules here by using the root user, but hey, that's what AWS says! Log in to your AWS account using your root credentials. Once logged in, select the Billing & Cost Management option highlighted under your account's name, as shown in the following screenshot:

This will pop up your account's management dashboard, using which you can view your account's Bills, set new Payment Methods, view past Payment History, and so on so forth. For now, select the Preferences option from the navigation page to bring up the Preferences dashboard, as shown in the following screenshot:

Select the Receive Billing Alerts checkbox to enable monitoring of your account's usage. It's important to, however, note that once you enable this checkbox, there is no going back! You will not be able to uncheck this option afterward!

Click on the Save Preferences option to save your new settings and then select the Manage Billing Alerts link to bring up CloudWatch's Create Alarm wizard, as shown in the following screenshot. This option is available from the CloudWatch dashboard to billing in the N. Virginia region.

The wizard will walk you through some simple steps to configure your first billing alarm. Select the checkbox adjoining to the EstimatedCharges option and click on Next to continue with the process. You can optionally change the statistic and period; however, I have gone ahead with the default values which is Maximum and 6 Hours respectively.

Moving on to the final step of your Create Alarm wizard, provide a suitable Name and Description for your billing alarm, as shown in the following screenshot. Next, configure the threshold for your alarm by selecting the >= (greater than or equal to) option and providing a threshold monetary amount such as $2 or $200, whichever is applicable to you. You can even set the threshold to $0.01, which will notify you the moment you start going out of the free tier eligibility. In either case, the alarm will only trigger when the actual cost of usage exceeds the monetary threshold that you have set. You can verify the setting by looking at the Alarm Preview graph as well. The red line indicates the threshold value set by you, whereas the blue highlighted portion is your account's current estimate bill:

With the Alarm's threshold set, the final thing that you need to do is define what action the alarm must take when it is triggered. From the Notification section, fill out the required details, as mentioned in the following:

You can add multiple e-mail IDs in the Email list field by separating them with commas. Once done, click on Save Changes to complete the alarm's creation process.

The alarm will take a few seconds to change from the INSUFFICIENT state to OK, as shown in the following screenshot. This is normal behavior as the alarm generally takes a few seconds to gather the metric data and verify it against the set threshold value.

You can view additional details about your newly created alarm by simply selecting it and checking out the Details and History tab provided in the following:

Oh! And one very important thing I almost forgot to mention! There is a catch to creating alarms, specifically billing ones, using CloudWatch. Don't worry, it's nothing serious! It's just that by design, the billing metric data of your entire account, which includes all your regions and AWS services, is collected and stored only in the US East (N. Virginia) region. So if you want to create or update this billing alarm at a later stage, you will have to change your default operating region to US East (N. Virginia) and then view and edit the billing alarm as required. You can, however, create EC2, ELB, RDS, and other AWS services related alarms from any particular region that you are operating from.

With the billing alarm created, let's try out something even more exciting! In this section, we will be creating a simple alarm to monitor an instance's CPU utilization. If the CPU utilization breaches a certain threshold, say 75 percent, then the alarm will trigger an e-mail notification as well as perform an additional task such as stop the instance.

To begin with, AWS makes creating alarms a really simple and straightforward process. The easiest way to do this is by selecting your individual instances from the EC2 Management Dashboard and selecting the Monitoring tab, as shown in the following screenshot. Each instance is monitored on a five-minute interval by default. You can modify this behavior and set the time interval as low as one minute by selecting the Enable Detailed Monitoring option.

Each instance, by default, gets its own set of performance graphs as well, which can be viewed in the Monitoring tab. These graphs generally include and display important metric information such as CPU utilization, disk Read/Writes, bytes transferred in terms of network IO, and so on. You can expand on each of the graphs by simply selecting them. This gives you a much better and detailed view of your instance's performance, as shown in the following image:

This is an example of an enhanced graph view of the CPU utilization metric. The x axis displays the CPU utilization in percent whereas the y axis display the time as per the current period's settings. You can view the individual data points and their associated values by simply hovering over them on the graph. Alternatively, you can also switch between the Statistics, Time Range, and Period as per your requirements. Once you have viewed your instance's performances, you can create a simple alarm by selecting the Create Alarm option provided in the Monitoring tab. This method is great if you want to set alarms for your instances on an individual basis, alternatively you can use the CloudWatch dashboard as well.

To view the CloudWatch Dashboard, from the AWS Management Console's home page, select the CloudWatch option, as shown in the following screenshot:

This will bring up the CloudWatch dashboard for the particular region in which you are currently operating. The dashboard is divided into two sections, a navigation pane to the left that groups and lists out your alarms based on their current state, for example, ALARM, OK, or INSUFFICIENT. It also provides access to the CloudWatch Logs and Metrics, as shown in the following screenshot:

Let's go ahead and check out the steps required to create our very first instance-based alarm. To get started, select the Create Alarm option. This will bring up the Create Alarm wizard, as shown in the following screenshot. The wizard is a simple two-step process that will help you with the necessary steps required to create your alarm.

First up, we need to select the correct metric that needs to be monitored. You can use the Browse Metrics or the search bar to filter out the particular metric, which in this case is CPU Utilization.

Next, select the particular instance for which you want to set this alarm. You can select multiple instances here as well. Selecting the instance will view its CPU utilization graph which you can modify using the statistics (Average) as well as the period (5 Minutes) dropdown lists. For now, click on Next to continue with the wizard.

The second step of the wizard is where you actually define the alarm, including its threshold value, as well as what actions have to be performed in case the alarm is triggered. For starters, provide a suitable Name and Description for your alarm. In this case, I provided the alarm with a name US-WEST-PROD-WEBSERVER-CPU —now that's pretty self-explanatory!

Moving on, the next part of your alarm's configuration is the threshold setting. As per our scenario, this alarm has to be triggered when the CPU utilization of the instance breaches 75 percent. Select the >= (greater than equal to) option from the is dropdown list and provide the value 75 in its adjoining textbox, as shown in the following screenshot. You can check your alarm's threshold settings in the Alarm Preview box.

With the threshold value set, the final thing to do is create the actions that will get triggered when the alarm is raised. There are three basic action items that you can create for each of your EC2 alarms described as follows:

For this particular scenario, we need to generate an e-mail-based notification when the alarm is raised and perform an EC2 action on the instance as well. Let's first create the notification action.

In the Notification section, select the option State is ALARM from the Whenever this alarm dropdown list. Next, click on the new list option to create a new SNS topic. Provide a suitable SNS Topic name is the Send notification to text field along with a list of comma separated e-mail addresses in the Email list, as shown in the following screenshot:

When the alarm is triggered, you will start receiving e-mails on the supplied e-mail list stating the nature of the alarm as well as the instance's metric data points at that particular time. You will receive these mails as long as the threshold value is breached or until the alarm state changes back to OK.

To add an additional EC2 action to this alarm, simply click on the +EC2 Action option. Follow the same process of setting the alarm's trigger state by selecting the State is ALARM option, as shown in the following screenshot. Next, select the particular EC2 action that you want to perform on this alarm's breach. In this case, I have opted for the instance to be stopped by selecting the Stop this instance option. Do remember that stopping an instance is only possible when your instances are backed by EBS volumes!

Now, here's something new for you. You will need to assign an IAM Role that will basically allow AWS to perform the EC2 actions on your instance. The alarm will auto-create an EC2ActionsAllow IAM Role for your convenience. The following is the code sample of the IAM Role for your reference. You can optionally create and assign your own IAM Roles as well use the IAM Management Dashboard (Chapter 2, Security and Access Management); however, this basic role should suffice for the time:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:Describe*",
        "ec2:Describe*",
        "ec2:RebootInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances"
      ],
      "Resource": "*"
    }
  ]
}

Select the Create IAM Role checkbox and verify the newly created IAM role. You can optionally even create additional action items that can get triggered when the alarm's threshold value changes to OK. Simply select the +Notification option and provide the details, as shown in the following screenshot. Remember, you can only create up to five actions for each alarm, so use them wisely!

Click on Create Alarm to complete the alarm's creation process. You can test your alarm's functionality by generating CPU load on your instance using a variety of tools such as Stress (http://people.seas.harvard.edu/~apw/stress/), Lookbusy (https://www.devin.com/lookbusy/), an so on. I personally use Lookbusy to generate artificial loads on my instances as it's pretty straightforward and easy to use. Do remember that these tools should only be used for testing and in no way are these tools recommended to be deployed on production workloads or instances. With this basic alarm created and tested, go ahead and create similar alarms for monitoring your instances disk as well as network utilization and performance!

Although CloudWatch does an excellent job at monitoring your instance's performance and status, it still has a few short comings to it. For starters, CloudWatch monitors your instance's CPU utilization, but cannot measure its load. Similarly, it can monitor the instance's memory size or a disk's IO performance, but cannot tell you the exact memory usage or the disk usage of a particular partition or layout. Why not? Well, simply because CloudWatch gets its monitoring metrics directly from the Xen hypervisors which host your instances. As a result, you don't get to see the performance and utilization of your instances at a very granular level. Luckily, CloudWatch is designed to accept metric values from other sources as well as from the hypervisor. These metrics are called as Custom Metrics and can be pushed into CloudWatch using a variety of ways. In this section, we are going to send custom metrics to CloudWatch using a set of simple Perl scripts provided by CloudWatch itself. These scripts have to be installed in your instance and are designed to send metric data periodically to CloudWatch. But before you begin, let's go through a few necessary prerequisite steps as follows.

Creating CloudWatch access roles

Just as with the alarm actions, the instances need to be provided with a special set of permissions to write to CloudWatch. There are two ways to go about this. The first method is to copy your secret and access keys to your instance, which let's face it is not the best of options! The second method is to create a role using IAM and assign your instances that role during their launch. The role will provide the instance with the necessary access rights to CloudWatch without having to expose any of your keys.

So, let's go ahead and create a simple access role for our instance using the IAM Management Dashboard.

From the IAM Management Dashboard, select Roles from the navigation pane. Next, select the option Create New Role. Provide a suitable Role Name for your new role and select Next Step to continue with the process:

Next up, from the Select Role Type page, select the Amazon EC2 option. This will bring up the Attach Policy page, as shown in the following screenshot. Using the Filter, search and select the CloudWatchFulllAccess policy as shown. You can alternatively create your very own custom CloudWatch access policy and attach that to your role if you want or use this default policy, which is the easier of the two. Click on Next Step to proceed with the wizard.

Review your role's information and finally select the Create Role option to complete the process.

Once your role is defined, go ahead and launch a new instance using the EC2 Management dashboard. Remember to assign this new role to your instance using the IAM role dropdown list, as shown in the following screenshot :

Note

A role is only assigned to an instance during its launch phase. You cannot assign roles to instances that are already running. To know more about IAM roles, refer to http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html.

With your instance launched, SSH into it using any of the options discussed in Chapter 3, Images and Instances. You are now ready to go ahead and install the necessary CloudWatch scripts!

Installing the CloudWatch monitoring scripts

Installing the CloudWatch scripts is a fairly straightforward process. The Perl scripts report an instance's memory, swap, and disk utilization metrics to CloudWatch. You can run these scripts off any Linux operating system, including the Amazon Linux AMI as well.

Run the following command in your instance's terminal to install and configure certain pre-requisite software:

# sudo yum install perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https 

Once completed, download the latest copy of the CloudWatch monitoring scripts using the following command:

# wget http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip

Note

The current version of the CloudWatch monitoring scripts is 1.2.1.

Next, unzip the contents of the downloaded Zip file using the following command:

# unzip CloudWatchMonitoringScripts-1.2.1.zip
# cd aws-scripts-mon

The output of the preceding commands is as follows:

The following are some of the important files that the CloudWatch monitoring script ZIP contains:

• CloudWatchClient.pm: This is a shared Perl module file that is used to make remote procedure calls to Amazon CloudWatch from other scripts.
• mon-put-instance-data.pl: This Perl script is responsible for collecting your instance's metrics (memory, swap, disk space utilization) and sending them to Amazon CloudWatch for processing.
• mon-get-instance-stats.pl: This Perl script is used to query CloudWatch and display the most recent utilization statistics for the EC2 instance on which this script is executed.
• awscreds.template: This file is used to store your AWS Secret and Access Keys. We will not be requiring this file as we have opted to use an IAM Role instead.

With this basic understanding in mind, let's use the mon-put-instance-data.pl script to view the instance's memory utilization (mem-util). Run the following command as shown here:

# ./mon-put-instance-data.pl --mem-util --verify --verbose  

Note that this command will not publish any metrics to CloudWatch because of the --verify attribute. Instead, it will only output the instance's memory utilization on the terminal, as shown in the following:

The script will initially search for the presence of the secret and access keys in the awscreds.template file. Since we have not provided the keys explicitly in the instance, the script then resorts to using the IAM role which we created earlier. Make sure that you receive confirmation from the script of its successful verification before you move on to the next steps.

Note

You can additionally use the --mem-used (memory used) and --mem-avail (memory available) metrics to query your instance's memory performance as well.

Next, run the following command to collect all your instance's memory related metrics and send them to CloudWatch:

# ./mon-put-instance-data.pl --mem-util --mem-used --mem-avail

You can even create a cron job and schedule the mon-put-instance-data.pl script to collect and send metric data over a period of time using the following set of commands. First, create a new file and save the following cron task in it:

# vi /etc/cron.d/Monitor_MEM

Add the following lines to your cron file:

*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --mem-used --mem-avail --from-cron

The cron will execute every five minutes and send the instance's memory details over to CloudWatch:

You can create additional cron files to monitor the instance's disk utilization as well. For example, if you want to be notified when the instance's root (/) or /var partition starts to fill up. In that case, create a simple cron task with the following information in it:

*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --disk-space-avail --disk-path=/ --disk-path=/var --from-cron

You can optionally even use the --disk-space-util (disk utilization) and the --disk-space-used (disk space used) metrics to query your instance's disk performance as well.

Viewing the custom metrics from CloudWatch

You can view your custom metrics from the CloudWatch management dashboard as well. Simply select the Metrics option from the CloudWatch navigation pane. Next, browse the listed metrics for a Linux System metrics, as shown in the following screenshot. You can optionally even use the Browse Metrics search bar to filter out the required metrics.

Select the Linux System Metrics option to view your instance's memory and disk utilizations, as shown in the following screenshot. You can use these metrics to list and create your very own custom alarms, as well use the Create Alarm option.

For example, raise an alarm when Memory Utilization of the instance crosses a threshold of 75 percent, or send a notification alert to the concerned sysadmins when the root (/) partition's available disk space is below 10 percent, and so on. With this, we have now successfully started monitoring our instance's memory and disk utilizations as well! Next up, let's look at how you can leverage CloudWatch to monitor your instance's or your application's log files using CloudWatch Logs!

Imagine that you have a bunch of web server instances with some web applications running on top of them. Now, what if you wanted to collect the log files off these instances and the web app and store it in a central repository such that you can troubleshoot errors and faults more effectively? That's precisely what CloudWatch Logs is all about!

CloudWatch Logs basically allows you to monitor custom application log files as well as log files generated by your EC2 instances in real time. You can even create and associate CloudWatch alarms which can send you notifications in case a particular log file displays errors. For example, you can monitor your application's logs for NullPointerExceptions or even your classical 404 status codes provided by your Apache web servers log files. You can additionally even store your log files to S3 for further analysis or to be loaded into some other log processing system.

In this section, we will learn how to monitor our application's web server (Apache HTTP) logs using CloudWatch Logs; however, before we proceed with that, let's first have a look at some of CloudWatch Log's concepts and terminologies.

Getting Started with CloudWatch Logs

CloudWatch Logs can be accessed from the CloudWatch management dashboard itself. Select the Logs option from the navigation pane to bring up the CloudWatch Logs dashboard. From the main page, select the option Create Log Group, as shown in the following screenshot:

In this scenario, we are going to monitor the HTTP logs of our web server instance, so go ahead and provide a suitable name for the log group, as shown in the following. Click on Create Log Group when done. In this case, I have named my log group as HTTP_LOG_GROUP.

With the log group now created, the next step is to create a log stream associated with it. From the log groups dashboard, select the name of your newly created log group. Here, you can create different log streams for your applications or OSs based on your requirements. Next, click on the Create Log Stream option. Provide a suitable name for your log stream and select the Create Log Stream button to complete the process. In this case, I have named my log stream as HTTP_LOG_STREAM, as shown in the following screenshot:

With these basic steps out of the way, now comes the fun part where we actually get to install and configure the log agent on the instance. To do so, launch a new Linux-based instance (I would recommend the Amazon Linux AMI or the Private Web Server AMI that we created in Chapter 4, Security, Storage, Networking, and Lots More!) and associate the CloudWatch access role with it. SSH into the instance and type in the following command to install the log agent:

# sudo yum install awslogs

With the log agent now installed, there are just a couple of files that you need to edit in order for the agent to work. The first file is the awscli.conf file. Open the file using any text editor of your choice and in the [default] section and specify the region where you want to view the log data. Since I am operating my instance out of the US-WEST (Oregon) region, I have provided us-west-2 as the region of my choice. You can provide either of these values as per your requirements: us-east-1, us-west-1, us-west-2, eu-west-1, eu-central-1, ap-southeast-1, ap-southeast-2, or ap-northeast-1. Now, run the following command:

# vi /etc/awslogs/awscli.conf

Once done, save the file and exit the editor:

Note

You can optionally provide your AWS secret key and access key information in the awscli.conf file; however, we have not done that as our instance is already associated with an IAM role.

The next file that you will need to edit is the awslogs.conf file. This is the primary log agent configuration file, using which you can define one or more log streams as well as define the logs that you want to track. Open the file using any text editor of your choice and paste the following lines toward the end of the file:

# vi /etc/awslogs/awslogs.conf
[/etc/httpd/logs/access_log]
file = /etc/httpd/logs/access_log
datetime_format = %b %d %H:%M:%S
initial_position = start_of_file
log_group_name = HTTP_LOG_GROUP
log_stream_name = HTTP_LOG_STREAM

What do all these lines mean? Here's a quick look at the awslogs.conf file's parameters:

• file: The file parameter specifies the log file whose contents you want to push on to the CloudWatch Logs. In my case, I want to specify the HTTP web server's access log file, hence the path of the Apache web server's access log file (/etc/httpd/logs/access_log).
• datetime_format: This parameter specifies how the timestamp value is extracted from supplied log file.
  • %b specifies month as in Jan, Feb, and so on.
  • %d specifies day of month in numbers as in 1,2,3,…31.
  • %H specifies hour in a 24 hour clock format.
  • %M specifies minutes.
  • %S specifies seconds.

  The access_log file's log entries have the timestamp of %b %d %H:%M:%S which translates to Sep 9 18:45:59.

• Initial_position: This parameter specifies where to start to read the log data from the log file. It supports two values: start_of_file and end_of_file.
• log_group_name: As the name implies, this parameter refers to the log group name that you created in CloudWatch Logs earlier.
• log_stream_name: This parameter refers to the destination log stream name.

Besides these standard parameters, the awslogs.conf file supports a few additional parameters as well. A complete list can be found at http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/AgentReference.html.

With the necessary entry made in the awslogs.conf file, we are now ready to start the log agent. Type in the following command in your instance's terminal screen to start the log agent service:

# sudo service awslogs start
# sudo chkconfig awslogs on

Viewing the logs

With the log agent up and running, you can now go ahead and view the HTTP logs from the CloudWatch Logs UI. Select the particular log group for which you want to view the log data. Next, select the appropriate log stream for the same. In this case, I had to select HTTP_LOG_GROUP as my log group and HTTP_LOG_STREAM as its associated log stream. You should see a bunch of log statements, as shown in the following screenshot:

You can use the Filter option to search for particular errors or status events from your log data. Alternatively, you can also use the Date/Time adjustor to view log data from a particular time period. With this step, we are now ready to go ahead and create a few metric filters for our log data.

Creating metric filters and alarms

CloudWatch Logs provide a really awesome method, using which you can filter and search out patterns, phrases, and even values from your log data. For example, you can create and set a filter that will raise an alarm when it encounters the words FATAL or ERROR from your application logs or even create a filter that searches for any 4XX-based errors from your HTTP logs such as 400 - Bad Request, 401 - Unauthorized, 403 - Forbidden, 404 - Not Found, and so on. Each time any of these values are found, CloudWatch registers them as a metric value, which can then be compared with the rest of the log data.

To create a metric filter, select your log group's name from the CloudWatch Log dashboard and select the Create Metric Filter option. This will bring up the Metric Filter wizard, as shown in the following screenshot:

Provide a suitable pattern to filter your log stream in the Filter Pattern field. In my case, I have created a simple filter that will extract the host, logName, user, request, size, and the status_code option if the status code has any 4XX values in it, which includes 401,403, 404, and so on.

Tip

You can read more about filter patterns and how to use them at http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html.

Next, select the correct log stream on which you wish to test this filter pattern. In my case, I selected the HTTP_LOG_STREAM option. Next, select the Test Pattern option to test your filter pattern. You should see a few results show up in the Results section if your filter pattern is accurate. This validates that the filter patter is correct, so move on to the next phase of the wizard where we assign a metric to this filter. Click on Assign Metric to continue. You should see the Create Metric Filter and Assign a Metric page, as shown in the following screenshot:

Using this page, you can assign a metric to your filter that can be used to graph as well as set alarms. Provide a suitable Filter Name for your newly created filter pattern. In my case, I have used the default values itself. Next, in the Metric Details section, provide an appropriate Metric Namespace, Metric Name, as well as a Metric Value for your filter. Click on Create Filter to complete the metric filter creation process. You should receive a confirmation box, as shown in the following screenshot. Click on Create Alarm to create and assign an alarm to this newly created metric filter.

That's all there is to it! You can refer to some interesting and easy to follow metric filter examples by following http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/MonitoringPolicyExamples.html.