Autoscaling on Amazon EC2

Amazon’s Auto Scaling service lets you launch or terminate EC2 instances (up to a defined minimum and maximum, respectively) automatically based on user-defined policies, schedules, and health checks. You can use Amazon’s CloudWatch for real-time monitoring of EC2 instances. Metrics such as CPU utilization, latency, and request counts are provided automatically by CloudWatch. Further, you can use CloudWatch to access up-to-the-minute statistics, view graphs, and set alarms (defined here):

Definition 1

An Amazon CloudWatch alarm is an object that watches over a single metric. An alarm can change state depending on the value of the metric. An action is invoked when an alarm changes state and remains in that state for a number of time periods.

You can configure a CloudWatch alarm to send a message to autoscaling whenever a specific metric has reached a threshold value. When the alarm sends the message, autoscaling executes the associated policy on an ASG to scale the group up or down. Note that an Auto Scaling action is invoked when the specified metric remains above the threshold value for a number of time periods. This is to ensure that a scaling action is not triggered due to a sudden spike in the metric.

Definition 2

A policy is a set of instructions for Auto Scaling that instructs the service how to respond to CloudWatch alarm messages.

Separate policies are instituted for autoscaling up and autoscaling down. The two key parameters associated with an Auto Scaling Policy are the following:

• ScalingAdjustment: The number of instances by which to scale. AdjustmentType determines the interpretation of this number (e.g., as an absolute number or as a percentage of the existing ASG size). A positive increment adds to the current capacity and a negative value removes from the current capacity.

• AdjustmentType: This specifies whether the ScalingAdjustment is an absolute number or a percentage of the current capacity. Valid values are ChangeIn​Capacity or PercentChangeInCapacity (described later in the chapter).

An autoscaling action, say scale up, usually takes a while to take effect. In light of this, you can specify a cooldown period (defined momentarily) to ensure that a new autoscaling event is triggered after the completion of the previous autoscaling event.

Definition 3

Cooldown is the period of time after autoscaling initiates a scaling activity during which no other scaling activity can take place. A cooldown period allows the effect of a scaling activity to become visible in the metrics that originally triggered the activity. This period is configurable and gives the system time to perform and adjust to any new scaling activities (such as scale-in and scale-out) that affect capacity.

On AWS, autoscaling also can be carried in a temporal fashion, referred to as scheduled scaling. In particular, scaling based on a schedule allows you to scale an application in response to predictable load changes. For instance, if the traffic begins to increase on Friday evening and remains high until Sunday evening, you can scale activities based on the predictable traffic patterns of the web application. To create a scheduled scaling action, you must specify the start time of the scaling action and the new minimum, maximum, and desired sizes for the scaling action. At the specified time, autoscaling updates the group with the values for minimum, maximum, and desired size specified by the scaling action.

At Netflix, we employed scaling by policy wherein a given cluster was scaled up/down based on the incoming request per second (RPS) of a given application. We used incoming RPS as the metric to drive autoscaling because it is independent of the application and directly relates to throughput.

Design Guidelines

In this section, we detail the various design guidelines underlying the algorithms for autoscaling discussed later in this chapter.

Avoiding the ping-pong effect

During a scale up event, new nodes are added to a given ASG. As a consequence, the RPS per node drops. However, if the RPS per node drops below the threshold specified for scaling down an ASG, it would trigger a scale-down event. This would result in alternating scale-up and scale-down events, as illustrated in Figure 6-4 (a), and referred to as a ping-pong effect. At Netflix, we observed that ping-ponging can potentially result in higher latency and, in the worst case, can cause violation of the Service-Level Agreement (SLA) of the service.

Thus, when defining the autoscaling policies, it is imperative to ensure that the policy is not susceptible to ping-pong effect. The desired autoscaling profile is exemplified by Figure 6-4 (b).

Figure 6-4. (a) Illustrating ping-pong effect (b) Desired autoscaling profile (Y-axis corresponds to the number of nodes in the ASG and X-axis corresponds to time)

Be proactive, not reactive

As mentioned earlier, applications such as the recommendation engine at Netflix take a long time to start. This can be ascribed to a variety of reasons; for example, loading of metadata of Netflix subscribers and precomputation of certain features. For such applications, it is critical to trigger the scale-up event in a proactive fashion, not reactively.

Let us consider the scenario shown in Figure 6-5. The solid arrow in the figure corresponds to the need to scale-up a given ASG as mandated by the SLA and increasing traffic. However, owing to a long application startup time, the autoscaling up is triggered, signified by the dashed arrow in the figure, proactively. The proactive approach ensures that the ASG is sufficiently provisioned by the time the latency approaches the SLA and that the SLA is never violated!

Figure 6-5. Illustration of scaling in a proactive fashion. Solid arrow signifies the need to scale up (as governed by SLA of the application at hand) and the dashed arrow signifies the corresponding autoscaling event (governed by the start up of the application)

Scalability Analysis

Determining the threshold for scale-up is an integral step is defining an autoscaling policy. A low threshold will result in under-utilization of the instances in the ASG; conversely, a high threshold can result in higher latency, thereby degrading the user experience. To this end, load testing is carried out to determine the throughput corresponding to the SLA of application (see Figure 6-6).

Figure 6-6. Trade-off between latency and throughput (load)

Properties

Each scale-up event should satisfy the following:

Property 1

RPS per node after scale up should be more than the scale-down threshold (T_D).

Property 1 ensures that scale-up does not induce a ping-pong effect. Likewise, each scale-down event should satisfy the following:

Property 2

RPS per node after scale-down should be less than the scale-up threshold (T_U).

Akin to Property 1, Property 2 also ensures that scale-up does not induce a ping-pong effect.

Autoscaling by Fixed Amount

In this section, we present a technique for scaling an ASG up/down by a fixed number of instances and as per the guidelines laid out earlier. AdjustmentType for the scaling policy is set to ChangeInCapacity, which is defined as follows:

ChangeInCapacity

This AdjustmentType is used to increase or decrease the capacity by a fixed amount on top of the existing capacity. For instance, let’s assume that the capacity of a given ASG is three and that ChangeInCapacity is set to five. When the policy is executed, autoscaling will add five more instances to ASG.

Algorithm 1 (shown in the following section) details the parameters and the steps to determine the scaling thresholds (for both scaling up and scaling down). The scale-down value D and the scale-up value U are inputs to the algorithm. The constants 0.90 and 0.50 used in defining T_U, T_D were determined empirically so as to minimize the impact on user experience and contain ASG under-utilization. Loop L1 in Algorithm 1 corresponds to scaling up an ASG as the incoming traffic increases. Loop L2 in Algorithm 1 scales down an ASG as the incoming traffic decreases.

Scaling by Percentage

In this section, we present  a technique for scaling an ASG up or down by a percentage of current capacity and as per the guidelines laid out earlier. AdjustmentType for the scaling policy is set to PercentChangeInCapacity, which is defined here:

PercentChangeInCapacity

This AdjustmentType is used to increase or decrease the capacity by a percentage of the desired capacity. For instance, let’s assume that an ASG has 15 instances and a scaling-up policy of the type PercentChangeInCapacity and adjustment set to 15. When the policy is run, autoscaling will increase the ASG size by two.

Note that if the PercentChangeInCapacity returns a value between 0 and 1, autoscaling will round it off to 1. If the PercentChangeInCapacity returns a value greater than 1, autoscaling will round it off to the lower value.

Algorithm 2 details the parameters and the steps to determine the scaling thresholds (for both scaling up and scaling down). The scale-down value D and the scale-up value U (note that both are percentages) are inputs to the algorithm. The constants 0.90 and 0.50 used in defining T_U and T_D were determined empirically so as to minimize impact on user experience and contain ASG under utilization. Loop L1 in Algorithm 2 corresponds to scaling up an ASG as the incoming traffic increases. Loop L2 in Algorithm 2 scales down an ASG as the incoming traffic decreases.

Startup Time Aware Scaling

In this section, we extend Algorithm 2 to guide autoscaling for applications with long startup times. Long application startup times can be ascribed to a variety of reasons; for example, loading of metadata. As discussed earlier, in the presence of long startup times, autoscaling up needs to be done proactively. For this, we employ the following steps:

• For a historical time–series of RPS in production, determine the change in RPS over every A_start minutes, where A_start denotes the application startup time. This would yield a time–series with these data points:

  • RPS_Astart − RPS₀

  • RPS_Astart+1 − RPS₁

  • RPS_Astart+2 − RPS₂

  • RPS_Astart+3 − RPS₃

  • ·

  • ·

  • ·

  where RPS_t denotes the RPS at time t. The derived time–series, referred to as rolling RPS change, captures the change in RPS in any window of width A_start minutes.

• Compute the 99th percentile of the rolling time–series, denoted by R_RPS.

• Compute γ = T_U − R_RPS. The parameter γ is the effective threshold for scale up. The use of 99th percentile of the rolling RPS change time–series is consistent with the Aggressive Upwards guideline outlined earlier.

The top of Figure 6-7 shows an example RPS time–series (with one-minute granularity) of an application. The startup time of the application was 30 minutes. The corresponding rolling RPS time–series is shown at the bottom of Figure 6-7. The 99th percentile of the rolling time–series is 1.109.

Figure 6-7. Rolling change in RPS_n for an application startup time of 30 minutes

Algorithm 3 details the parameters and the steps to determine the scaling thresholds (for both scaling up and scaling down). The scale-down value D and the scale-up value U are inputs to the algorithm. The constants 0.90 and 0.50 used in defining T_U, T_D were determined empirically so as to minimize the impact on user experience and contain ASG under-utilization. Loop L1 in “Algorithm 3” corresponds to scaling-up an ASG as the incoming traffic increases. Loop L2 in Algorithm 3 scales-down an ASG as the incoming traffic decreases. Unlike scale-up, the threshold for scale-down D need not be adjusted, because applications do not induce a long delay during termination of instances on Amazon’s EC2.

Potpourri

There have been cases wherein the CPU utilization on production nodes spiked without any increase in traffic. This can happen to a variety of accidental events. To handle such cases, instituting add-on scale-up policies (i.e., besides a scale-up policy based on RPS), as exemplified in Figure 6-8, helps to mitigate the impact on the end users.

Figure 6-8. Add-on policies to check “meltdown”

In July 2015, AWS introduced new scaling policies with steps. For example, you can specify different responses for different levels of average CPU utilization, say <50%, [50%, 60%), [60%, 80%), and ≥80%. Further, if you create multiple-step scaling policies for the same resource (perhaps based on CPU utilization and inbound network traffic) and both of them fire at approximately the same time, autoscaling will look at both policies and choose the one that results in the change of the highest magnitude.

In certain scenarios, you might want to protect certain instances in an ASG from termination. For example, an instance might be handling a long-running work task, perhaps pulled from an SQS queue. Protecting the instance from termination will avoid wasted work. In a similar vein, an instance might serve a special purpose within the group; for example, it could be the master node of a Hadoop cluster, or a “canary” that flags the entire group of instances as up and running. To this end, you can use the Instance Protection Feature offered by AWS. In most cases, at least one instance in an ASG should be left unprotected; if all of the instances are protected, no scale action will be taken.

Leading companies such as Netflix and Facebook have been using autoscaling to improve cluster performance, service availability, and reduce costs. In its post, Facebook shared the following:

...a particular type of web server at Facebook consumes about 60 watts of power when it’s idle (0 RPS, or requests-per-second). The power consumption jumps to 130 watts when it runs at low-level CPU utilization (small RPS). But when it runs at medium-level CPU utilization, power consumption increases only slightly to 150 watts. Therefore, from a power-efficiency perspective, we should try to avoid running a server at low RPS and instead try to run at medium RPS.

For details, refer to the following:

• http://techblog.netflix.com/2013/12/scryer-netflixs-predictive-auto-scaling.html

• http://bit.ly/fb-infrastructure-autoscale

Besides RPS, other metrics have been used for autoscaling: CPU utilization, memory usage, disk I/O bandwidth, network link load, peak workload, jobs in progress, service rate, the number of concurrent users, the number of active connections, jitter, delay, and the average response time per request. Regression modeling has been employed for predicting the amount of resources needed for the actual workload and possibly retract over-provisioned resources. Likewise, several other approaches have been employed for resource-demand prediction such as prediction based on changes in the request arrival rate (i.e., the slope of the workload). You can employ sensitivity analysis to characterize the different types of inputs and determine the types of resources that have the highest impact on the throughput (or the performance metric of interest) of the application. Subsequently, you can set up multiple autoscaling rules based on one or more resource types.

In recent years, the use of containers has received wide attention. Amazon EC2 Container Service (ECS), Google Container Engine, and Microsoft Azure Container Service are the most popular public container services. Multi-AZ clusters make the ECS infrastructure highly available, thereby providing a safeguard from potential zone failure. The AZ–aware ECS scheduler manages, scales, and distributes the tasks across the cluster, thus making the architecture highly available. On AWS, akin to the EC2 instances, autoscaling policies also can be defined for ECS instances. You can use the approaches discussed earlier in this chapter in the context of autoscaling container instances, as well. 

Advanced Approaches

Given the importance of exploiting the elasticity of the cloud in the best possible fashion, several advanced techniques have been proposed for autoscaling in both the industry and the academia. For instance, Facebook employed the classic control theory and proportional-integral (PI) controller to achieve fast reaction time. Netflix developed two prediction algorithms—one of which is an augmented linear regression–based algorithm, the other based on Fast Fourier Transform (FFT). One of the key highlights of the Netflix approach is that it’s predictive. Specifically, its approach learns the request pattern based on historical data and subsequently drives the scale-up or scale-down action. Both approaches have been deployed in production environments. Given that no comparative analysis was presented, it is difficult to assess how these techniques fare against the techniques that were proposed previously.

Many other approaches for autoscaling have been proposed based on control theory, queuing theory, fuzzy logic, neural networks, reinforcement learning, support vector machines, wavelet transform, regression splines, pattern matching, Kalman filters, sliding window, proportional thresholding, second-order regression, histograms, time–series models, the secant method, voting system, and look-ahead control. In most cases, these techniques “learn” from past traffic patterns and resource usage and hence are unable to adapt with any new pattern that might appear as a result of the dynamic nature of the web traffic.

In practice, applicability of autoscaling approaches based on the preceding is limited owing to a wide variety of reasons. For instance, reinforcement learning–based approaches require a long time to learn and adapt only to slowly changing conditions. Therefore, you cannot apply such techniques to real applications that usually experience sudden traffic bursts. In a similar vein, queuing theory–based approaches impose hard assumptions that are typically not valid for real, complex systems. Besides, such approaches are intended for stationary scenarios and hence you will need to recalculate the queuing model when the conditions of the application change. In the case of control theory–based approaches, determining the gain parameters is nontrivial.