Chapter 2. Cloud Controller Design
OpenStack is designed to be massively horizontally scalable, which allows all services to be distributed widely. However, to simplify this guide we have decided to discuss services of a more central nature using the concept of a single cloud controller.
For more details about the overall architecture, see the Chapter 7.
As described in this guide, the cloud controller is a single node that hosts the databases, message queue service, authentication and authorization service, image management service, user dashboard, and API endpoints.
The cloud controller provides the central management system for multi-node OpenStack deployments. Typically the cloud controller manages authentication and sends messaging to all the systems through a message queue.
For our example, the cloud controller has a collection of
nova-* components that represent the global
state of the cloud, talks to services such as authentication,
maintains information about the cloud in a database,
communicates to all compute nodes and storage
workers through a queue, and
provides API access. Each service running on a designated
cloud controller may be broken out into separate nodes for
scalability or availability.
Hardware Considerations
A cloud controller’s hardware can be the same as a compute node, though you may want to further specify based on the size and type of cloud that you run.
It’s also possible to use virtual machines for all or some of the services that the cloud controller manages, such as the message queuing. In this guide, we assume that all services are running directly on the cloud controller.
To size the server correctly, and determine whether to virtualize any part of it, you should estimate:
-
The number of instances that you expect to run
-
The number of compute nodes that you have
-
The number of users who will access the compute or storage services
-
Whether users interact with your cloud through the REST API or the dashboard
-
Whether users authenticate against an external system (such as, LDAP or Active Directory)
-
How long you expect single instances to run
| Consideration | Ramification |
|---|---|
|
How many instances will run at once? |
Size your database server accordingly, and scale out beyond one cloud controller if many instances will report status at the same time and scheduling where a new instance starts up needs computing power. |
|
How many compute nodes will run at once? |
Ensure that your messaging queue handles requests successfully and size accordingly. |
|
How many users will access the API? |
If many users will make multiple requests, make sure that the CPU load for the cloud controller can handle it. |
|
How many users will access the dashboard? |
The dashboard makes many requests, even more than the API access, so add even more CPU if your dashboard is the main interface for your users. |
|
How many |
You need to size the controller with a core per service. |
|
How long does a single instance run? |
Starting instances and deleting instances is demanding on the compute node but also demanding on the controller node because of all the API queries and scheduling needs. |
|
Does your authentication system also verify externally? |
Ensure network connectivity between the cloud controller and external authentication system are good and that the cloud controller has the CPU power to keep up with requests. |
Separation of Services
While our example contains all central services in a single location, it is possible and indeed often a good idea to separate services onto different physical servers. The following is a list of deployment scenarios we’ve seen, and their justifications.
|
Run |
This deployment felt the spare I/O on the Object Storage proxy server was sufficient, and the Image Delivery portion of Glance benefited from being on physical hardware and having good connectivity to the Object Storage back-end it was using. |
|
Run a central dedicated database server |
This deployment made a central dedicated server to provide the databases for all services. This simplified operations by isolating database server updates, and allowed for the simple creation of slave database servers for failover. |
|
Run one VM per service |
This deployment ran central services on a set of servers running KVM. A dedicated VM was created for each service (nova-scheduler, rabbitmq, database etc). This assisted the deployment with scaling as they could tune the resources given to each virtual machine based on the load they received (something that was not well understood during installation). |
|
Use an external load balancer |
This deployment had an expensive
hardware load balancer in their
organisation. They ran multiple
|
One choice that always comes up is whether to virtualize or not. Some services, such as nova-compute, swift-proxy and swift-object servers, should not be virtualized. However, control servers can often be happily virtualized - the performance penalty can usually be offset by simply running more of the service.
Database
Most OpenStack Compute central services, and currently also the nova-compute nodes, use the database for stateful information. Loss of this ability leads to errors. As a result, we recommend that you cluster your databases in some way to make them failure tolerant.
Message Queue
Most OpenStack Compute services communicate with each other using the Message Queue. In general, if the message queue fails or becomes inaccessible, the cluster grinds to a halt and ends up in a “read only” state, with information stuck at the point where the last message was sent. Accordingly, we recommend that you cluster the message queue - and RabbitMQ has in-build abilities to do this.
Application Programming Interface (API)
All public access, whether direct, through a command line client, or through the web-based dashboard, uses the API service. Find the API reference at http://api.openstack.org/.
You must choose whether you want to support the Amazon EC2 compatibility APIs, or just the OpenStack APIs. One issue you might encounter when running both APIs is an inconsistent experience when referring to images and instances.
For example, the EC2 API refers to instances using IDs that contain hexadecimal whereas the OpenStack API uses names and digits. Similarly, the EC2 API tends to rely on DNS aliases for contacting virtual machines, as opposed to OpenStack which typically lists IP addresses.
If OpenStack is not set up in the right way, it is simple to have scenarios where users are unable to contact their instances due to only having an incorrect DNS alias. Despite this, EC2 compatibility can assist users migrating to your cloud.
Like databases and message queues, having more than one
API server is a good thing.
Traditional HTTP load balancing techniques can be used to
achieve a highly available nova-api
service.
Extensions
The API Specifications (http://docs.openstack.org/api/api-specs.html) define the core actions, capabilities, and media-types of the OpenStack API. A client can always depend on the availability of this core API and implementers are always required to support it in its entirety. Requiring strict adherence to the core API allows clients to rely upon a minimal level of functionality when interacting with multiple implementations of the same API.
The OpenStack Compute API is extensible. An extension adds capabilities to an API beyond those defined in the core. The introduction of new features, MIME types, actions, states, headers, parameters, and resources can all be accomplished by means of extensions to the core API. This allows the introduction of new features in the API without requiring a version change and allows the introduction of vendor-specific niche functionality.
Scheduler
Fitting various sized virtual machines (different flavors) into different sized physical nova-compute nodes is a challenging problem - researched generically in Computer Science as a packing problem.
You can use various techniques to handle this problem, one of which is to have flavor sizes scale linearly, be a proportional fraction of your physical node capacity, though solving this problem is out of the scope of this book. To support your scheduling choices, OpenStack Compute provides several different types of scheduling drivers, a full discussion of which is found in the reference manual (http://docs.openstack.org/folsom/openstack-compute/admin/content/ch_scheduling.html).
For availability purposes, or for very large or high-schedule frequency installations, you should consider running multiple nova-scheduler services. No special load balancing is required, as the nova-scheduler communicates entirely using the message queue.
Images
The OpenStack Image Catalog and Delivery service
consists of two parts - glance-api and
glance-registry. The former is
responsible for the delivery of images and the compute
node uses it to download images from the back-end. The
latter maintains the metadata information associated with
virtual machine images and requires a database.
The glance-api part is an abstraction layer
that allows a choice of back-end. Currently, it
supports:
-
OpenStack Object Storage. Allows you to store images as objects.
-
File system. Uses any traditional file system to store the images as files.
-
S3. Allows you to fetch images from Amazon S3.
-
HTTP. Allows you to fetch images from a web server. You cannot write images by using this mode.
If you have an OpenStack Object Storage service, we recommend using this as a scalable place to store your images. You can also use a file system with sufficient performance or Amazon S3 - unless you do not need the ability to upload new images through OpenStack.
Dashboard
The OpenStack Dashboard is implemented as a Python web
application that runs in Apache
httpd. Therefore, you may treat it the same
as any other web application, provided it can reach the
API servers (including their admin endpoints) over the
network.
Authentication and Authorization
The concepts supporting OpenStack’s authentication and authorization are derived from well understood and widely used systems of a similar nature. Users have credentials they can use to authenticate, and they can be a member of one or more groups (known as projects or tenants interchangeably).
For example, a cloud administrator might be able to list all instances in the cloud, whereas a user can only see those in their current group. Resources quotas, such as the number of cores that can be used, disk space, and so on, are associated with a project.
The OpenStack Identity Service (Keystone) is the point that provides the authentication decisions and user attribute information, which is then used by the other OpenStack services to perform authorization. Policy is set in the policy.json file. For information on how to configure these, see Chapter 9.
The Identity Service supports different plugins for back-end authentication decisions, and storing information. These range from pure storage choices to external systems and currently include:
-
In-memory Key-Value Store
-
SQL database
-
PAM
-
LDAP
Many deployments use the SQL database, however LDAP is also a popular choice for those with existing authentication infrastructure that needs to be integrated.
Network Considerations
Because the cloud controller handles so many different services, it must be able to handle the amount of traffic that hits it. For example, if you choose to host the OpenStack Imaging Service on the cloud controller, the cloud controller should be able to support the transferring of the images at an acceptable speed.
As another example, if you choose to use single-host networking where the cloud controller is the network gateway for all instances, then the Cloud Controller must support the total amount of traffic that travels between your cloud and the public Internet.
We recommend that you use a fast NIC, such as 10 GB. You can also choose to use two 10 GB NICs and bond them together. While you might not be able to get a full bonded 20 GB speed, different transmission streams use different NICs. For example, if the Cloud Controller transfers two images, each image uses a different NIC and gets a full 10 GB of bandwidth.