If your cloud is successful, eventually you must add resources to meet the increasing demand. OpenStack is designed to be horizontally scalable. Rather than switching to larger servers, you procure more servers. Ideally, you scale out and load balance among functionally-identical services.
Determining the scalability of your cloud and how to improve it is an exercise with many variables to balance. No one solution meets everyone’s scalability aims. However, it is helpful to track a number of metrics.
The starting point for most is the core count of your
cloud. By applying some ratios, you can gather information
about the number of virtual machines (VMs) you expect to
run ((overcommit fraction × cores) / virtual cores
per instance)
, how much storage is required
(flavor disk size × number of instances)
.
You can use these ratios to determine how much additional
infrastructure you need to support your cloud.
The default OpenStack flavors are:
Name | Virtual cores | Memory | Disk | Ephemeral |
---|---|---|---|---|
m1.tiny |
1 |
512 MB |
1 GB |
0 GB |
m1.small |
1 |
2 GB |
10 GB |
20 GB |
m1.medium |
2 |
4 GB |
10 GB |
40 GB |
m1.large |
4 |
8 GB |
10 GB |
80 GB |
m1.xlarge |
8 |
16 GB |
10 GB |
160 GB |
Assume that the following set-up supports (200 / 2) × 16
= 1600 VM instances and requires 80 TB of storage for
/var/lib/nova/instances
:
200 physical cores
Most instances are size m1.medium (2 virtual cores, 50 GB of storage)
Default CPU over-commit ratio
(cpu_allocation_ratio
in
nova.conf) of 16:1
However, you need more than the core count alone to estimate the load that the API services, database servers, and queue servers are likely to encounter. You must also consider the usage patterns of your cloud.
As a specific example, compare a cloud that supports a managed web hosting platform with one running integration tests for a development project that creates one VM per code commit. In the former, the heavy work of creating a VM happens only every few months, whereas the latter puts constant heavy load on the cloud controller. You must consider your average VM lifetime, as a larger number generally means less load on the cloud controller.
Aside from the creation and termination of VMs, you must consider the impact of users accessing the service — particularly on nova-api and its associated database. Listing instances garners a great deal of information and, given the frequency with which users run this operation, a cloud with a large number of users can increase the load significantly. This can even occur without their knowledge — leaving the OpenStack Dashboard instances tab open in the browser refreshes the list of VMs every 30 seconds.
After you consider these factors, you can determine how many cloud controller cores you require. A typical 8 core, 8 GB of RAM server is sufficient for up to a rack of compute nodes — given the above caveats.
You must also consider key hardware specifications for the performance of user VMs. You must consider both budget and performance needs. Examples include: Storage performance (spindles/core), memory availability (RAM/core), network bandwidth (Gbps/core), and overall CPU performance (CPU/core).
For which metrics to track to determine how to scale your cloud, see Chapter 13.
You can facilitate the horizontal expansion of your cloud by adding nodes. Adding compute nodes is straightforward — they are easily picked up by the existing installation. However, you must consider some important points when you design your cluster to be highly available.
Recall that a cloud controller node runs several
different services. You can install services that
communicate only using the message queue internally
— nova-scheduler
and
nova-console
— on a new server for
expansion. However, other integral parts require more
care.
You should load balance user-facing services such as
Dashboard, nova-api
or the Object Storage
proxy. Use any standard HTTP load balancing method (DNS
round robin, hardware load balancer, software like Pound
or HAProxy). One caveat with Dashboard is the VNC proxy,
which uses the WebSocket protocol — something that a L7
load balancer might struggle with. See also Horizon session storage
(http://docs.openstack.org/developer/horizon/topics/deployment.html#session-storage).
You can configure some services, such as
nova-api
and glance-api
, to
use multiple processes by changing a flag in their
configuration file — allowing them to share work between
multiple cores on the one machine.
Several options are available for MySQL load balancing, and RabbitMQ has in-built clustering support. Information on how to configure these and many of the other services can be found in the Operations Section.
Use one of the following OpenStack methods to segregate your cloud: cells, regions, zones and host aggregates. Each method provides different functionality, as described in the following table:
Cells | Regions | Availability Zones | Host Aggregates | |
---|---|---|---|---|
Use when you need |
A single API endpoint for compute, or you require a second level of scheduling. |
Discrete regions with separate API endpoints and no coordination between regions. |
Logical separation within your nova deployment for physical isolation or redundancy. |
To schedule a group of hosts with common features. |
Example |
A cloud with multiple sites where you can schedule VMs “anywhere” or on a particular site. |
A cloud with multiple sites, where you schedule VMs to a particular site and you want a shared infrastructure. |
A single site cloud with equipment fed by separate power supplies. |
Scheduling to hosts with trusted hardware support. |
Overhead |
|
|
|
|
Shared services |
Keystone
|
Keystone |
Keystone All nova services |
Keystone All nova services |
This array of options can be best divided into two — those which result in running separate nova deployments (cells and regions), and those which merely divide a single deployment (availability zones and host aggregates).
OpenStack Compute cells are designed to allow
running the cloud in a distributed fashion without
having to use more complicated technologies, or being
invasive to existing nova installations. Hosts in a
cloud are partitioned into groups called
cells. Cells are configured
in a tree. The top-level cell (“API cell”) has a host
that runs the nova-api
service, but no
nova-compute
services. Each child
cell runs all of the other typical nova-*
services found in a regular installation, except for
the nova-api
service. Each cell has its
own message queue and database service, and also runs
nova-cells
— which manages the
communication between the API cell and child
cells.
This allows for a single API server being used to
control access to multiple cloud installations.
Introducing a second level of scheduling (the cell
selection), in addition to the regular
nova-scheduler
selection of hosts,
provides greater flexibility to control where virtual
machines are run.
Contrast this with regions. Regions have a separate API endpoint per installation, allowing for a more discrete separation. Users wishing to run instances across sites have to explicitly select a region. However, the additional complexity of a running a new service is not required.
The OpenStack Dashboard (Horizon) currently only uses a single region, so one dashboard service should be run per region. Regions are a robust way to share some infrastructure between OpenStack Compute installations, while allowing for a high degree of failure tolerance.
You can use availability zones, host aggregates, or both to partition a nova deployment.
Availability zones are implemented through and configured in a similar way to host aggregates.
However, you use an availability zone and a host aggregate for different reasons:
Availability zone. Enables you to arrange OpenStack Compute hosts into logical groups, and provides a form of physical isolation and redundancy from other availability zones, such as by using separate power supply or network equipment.
You define the availability zone in which a specified Compute host resides locally on each server. An availability zone is commonly used to identify a set of servers that have a common attribute. For instance, if some of the racks in your data center are on a separate power source, you can put servers in those racks in their own availability zone. Availability zones can also help separate different classes of hardware.
When users provision resources, they can specify from which availability zone they would like their instance to be built. This allows cloud consumers to ensure that their application resources are spread across disparate machines to achieve high availability in the event of hardware failure.
Host aggregate. Enables you to partition OpenStack Compute deployments into logical groups for load balancing and instance distribution. You can use host aggregates to further partition an availability zone. For example, you might use host aggregates to partition an availability zone into groups of hosts that either share common resources, such as storage and network, or have a special property, such as trusted computing hardware.
A common use of host aggregates is to provide information for use with the nova-scheduler. For example, you might use a host aggregate to group a set of hosts that share specific flavors or images.
Previously, all services had an availability zone. Currently, only the nova-compute service has its own availability zone. Services such as nova-scheduler, nova-network, nova-conductor have always spanned all availability zones.
When you run any of the following operations, the services appear in their own internal availability zone (CONF.internal_service_availability_zone):
nova host-list (os-hosts)
euca-describe-availability-zones verbose
nova-manage service list
The internal availability zone is hidden in euca-describe-availability_zones (non-verbose).
CONF.node_availability_zone has been renamed to CONF.default_availability_zone and is only used by the nova-api and nova-scheduler services.
CONF.node_availability_zone still works but is deprecated.
While several resources already exist to help with deploying and installing OpenStack, it’s very important to make sure you have your deployment planned out ahead of time. This guide expects at least a rack has been set aside for the OpenStack cloud but also offers suggestions for when and what to scale.
“The Cloud” has been described as a volatile environment where servers can be created and terminated at will. While this may be true, it does not mean that your servers must be volatile. Ensuring your cloud’s hardware is stable and configured correctly means your cloud environment remains up and running. Basically, put effort into creating a stable hardware environment so you can host a cloud that users may treat as unstable and volatile.
OpenStack can be deployed on any hardware supported by an OpenStack-compatible Linux distribution, such as Ubuntu 12.04 as used in this books’ reference architecture.
Hardware does not have to be consistent, but should at least have the same type of CPU to support instance migration.
The typical hardware recommended for use with OpenStack is the standard value-for-money offerings that most hardware vendors stock. It should be straightforward to divide your procurement into building blocks such as “compute,” “object storage,” and “cloud controller,” and request as many of these as desired. Alternately should you be unable to spend more, if you have existing servers, provided they meet your performance requirements and virtualization technology, these are quite likely to be able to support OpenStack.
OpenStack is designed to increase in size in a straightforward manner. Taking into account the considerations in the Scalability chapter — particularly on the sizing of the cloud controller — it should be possible to procure additional compute or object storage nodes as needed. New nodes do not need to be the same specification, or even vendor, as existing nodes.
For compute nodes, nova-scheduler
will
take care of differences in sizing to do with core
count and RAM amounts, however you should consider the
user experience changes with differing CPU speeds.
When adding object storage nodes, a
weight should be specified
that reflects the capability of
the node.
Monitoring the resource usage and user growth will enable you to know when to procure. The Monitoring chapter details some useful metrics.
Server hardware’s chance of failure is high at the start and the end of its life. As a result, much effort in dealing with hardware failures while in production can be avoided by appropriate burn-in testing to attempt to trigger the early-stage failures. The general principle is to stress the hardware to its limits. Examples of burn-in tests include running a CPU or disk benchmark for several days.