Cloud Controller

Compute Nodes

libvirt: /var/log/libvirt/libvirtd.log

Console (boot up messages) for VM instances: /var/lib/nova/instances/instance-<instance id>/console.log

Block Storage Nodes

cinder: /var/log/cinder/cinder-volume.log

rsyslog Client Configuration

To begin, configure all OpenStack components to log to syslog in addition to their standard log file location. Also configure each component to log to a different syslog facility. This makes it easier to split the logs into individual components on the central server.

nova.conf:

use_syslog=True
syslog_log_facility=LOG_LOCAL0

glance-api.conf and glance-registry.conf:

use_syslog=True
syslog_log_facility=LOG_LOCAL1

cinder.conf:

use_syslog=True
syslog_log_facility=LOG_LOCAL2

keystone.conf:

use_syslog=True
syslog_log_facility=LOG_LOCAL3

Swift

By default, Swift logs to syslog.

Next, create /etc/rsyslog.d/client.conf with the following line:

*.* @192.168.1.10

This instructs rsyslog to send all logs to the IP listed. In this example, the IP points to the Cloud Controller.

rsyslog Server Configuration

Designate a server as the central logging server. The best practice is to choose a server that is solely dedicated to this purpose. Create a file called /etc/rsyslog.d/server.conf with the following contents:

# Enable UDP 
$ModLoad imudp 
# Listen on 192.168.1.10 only 
$UDPServerAddress 192.168.1.10
# Port 514 
$UDPServerRun 514  
      
# Create logging templates for nova
$template NovaFile,"/var/log/rsyslog/%HOSTNAME%/nova.log" 
$template NovaAll,"/var/log/rsyslog/nova.log"
      

      
# Log everything else to syslog.log 
$template DynFile,"/var/log/rsyslog/%HOSTNAME%/syslog.log"
*.* ?DynFile
      

      
# Log various openstack components to their own individual file
local0.* ?NovaFile 
local0.* ?NovaAll 
& ~

The above example configuration handles the nova service only. It first configures rsyslog to act as a server that runs on port 514. Next, it creates a series of logging templates. Logging templates control where received logs are stored. Using the example above, a nova log from c01.example.com goes to the following locations:

• /var/log/rsyslog/c01.example.com/nova.log

• /var/log/rsyslog/nova.log

This is useful as logs from c02.example.com go to:

• /var/log/rsyslog/c02.example.com/nova.log

• /var/log/rsyslog/nova.log

So you have an individual log file for each compute node as well as an aggregated log that contains nova logs from all nodes.

Process Monitoring

A basic type of alert monitoring is to simply check and see if a required process is running. For example, ensure that the nova-api service is running on the Cloud Controller:

[ root@cloud ~ ] # ps aux | grep nova-api
nova 12786 0.0 0.0 37952 1312 ? Ss Feb11 0:00 su -s /bin/sh -c exec nova-api --config-file=/etc/nova/nova.conf nova
nova 12787 0.0 0.1 135764 57400 ? S Feb11 0:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova 12792 0.0 0.0 96052 22856 ? S Feb11 0:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova 12793 0.0 0.3 290688 115516 ? S Feb11 1:23 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova 12794 0.0 0.2 248636 77068 ? S Feb11 0:04 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
root 24121 0.0 0.0 11688 912 pts/5 S+ 13:07 0:00 grep nova-api

You can create automated alerts for critical processes by using Nagios and NRPE. For example, to ensure that the nova-compute process is running on compute nodes, create an alert on your Nagios server that looks like this:

define service { 
    host_name c01.example.com 
    check_command check_nrpe_1arg!check_nova-compute 
    use generic-service 
    notification_period 24x7 
    contact_groups sysadmins 
    service_description nova-compute 
}

Then on the actual compute node, create the following NRPE configuration:

command[check_nova-compute]=/usr/lib/nagios/plugins/check_procs -c 1: -a nova-compute

Nagios checks that at least one nova-compute service is running at all times.

Resource Alerting

Resource alerting provides notifications when one or more resources are critically low. While the monitoring thresholds should be tuned to your specific OpenStack environment, monitoring resource usage is not specific to OpenStack at all – any generic type of alert will work fine.

Some of the resources that you want to monitor include:

• Disk Usage

• Server Load

• Memory Usage

• Network IO

• Available vCPUs

For example, to monitor disk capacity on a compute node with Nagios, add the following to your Nagios configuration:

define service { 
    host_name c01.example.com 
    check_command check_nrpe!check_all_disks!20% 10% 
    use generic-service 
    contact_groups sysadmins 
    service_description Disk 
}

On the compute node, add the following to your NRPE configuration:

command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -e

Nagios alerts you with a WARNING when any disk on the compute node is 80% full and CRITICAL when 90% is full.

OpenStack-specific Resources

Resources such as memory, disk, and CPU are generic resources that all servers (even non-OpenStack servers) have and are important to the overall health of the server. When dealing with OpenStack specifically, these resources are important for a second reason: ensuring enough are available in order to launch instances. There are a few ways you can see OpenStack resource usage.

The first is through the nova command:

# nova usage-list

This command displays a list of how many instances a tenant has running and some light usage statistics about the combined instances. This command is useful for a quick overview of your cloud, but doesn’t really get into a lot of details.

Next, the nova database contains three tables that store usage information.

The nova.quotas and nova.quota_usages tables store quota information. If a tenant’s quota is different than the default quota settings, their quota is stored in nova.quotas table. For example:

mysql> select project_id, resource, hard_limit from quotas; 
+----------------------------------+-----------------------------+------------+
| project_id                       | resource                    | hard_limit |
+----------------------------------+-----------------------------+------------+
| 628df59f091142399e0689a2696f5baa | metadata_items              | 128        |
| 628df59f091142399e0689a2696f5baa | injected_file_content_bytes | 10240      |
| 628df59f091142399e0689a2696f5baa | injected_files              | 5          |
| 628df59f091142399e0689a2696f5baa | gigabytes                   | 1000       |
| 628df59f091142399e0689a2696f5baa | ram                         | 51200      |
| 628df59f091142399e0689a2696f5baa | floating_ips                | 10         |
| 628df59f091142399e0689a2696f5baa | instances                   | 10         |
| 628df59f091142399e0689a2696f5baa | volumes                     | 10         |
| 628df59f091142399e0689a2696f5baa | cores                       | 20         |
+----------------------------------+-----------------------------+------------+ 

The nova.quota_usages table keeps track of how many resources the tenant currently has in use:

mysql> select project_id, resource, in_use from quota_usages where project_id like '628%';
+----------------------------------+--------------+--------+ 
| project_id                       | resource     | in_use | 
+----------------------------------+--------------+--------+ 
| 628df59f091142399e0689a2696f5baa | instances    | 1      |
| 628df59f091142399e0689a2696f5baa | ram          | 512    | 
| 628df59f091142399e0689a2696f5baa | cores        | 1      | 
| 628df59f091142399e0689a2696f5baa | floating_ips | 1      | 
| 628df59f091142399e0689a2696f5baa | volumes      | 2      | 
| 628df59f091142399e0689a2696f5baa | gigabytes    | 12     | 
| 628df59f091142399e0689a2696f5baa | images       | 1      | 
+----------------------------------+--------------+--------+

By combining the resources used with the tenant’s quota, you can figure out a usage percentage. For example, if this tenant is using 1 Floating IP out of 10, then they are using 10% of their Floating IP quota. You can take this procedure and turn it into a formatted report:

+----------------------------------+------------+-------------+---------------+
| some_tenant                                                                 |
+-----------------------------------+------------+------------+---------------+
| Resource                          | Used       | Limit      |               |
+-----------------------------------+------------+------------+---------------+
| cores                             | 1          | 20         |           5 % |
| floating_ips                      | 1          | 10         |          10 % |
| gigabytes                         | 12         | 1000       |           1 % |
| images                            | 1          | 4          |          25 % |
| injected_file_content_bytes       | 0          | 10240      |           0 % |
| injected_file_path_bytes          | 0          | 255        |           0 % |
| injected_files                    | 0          | 5          |           0 % |
| instances                         | 1          | 10         |          10 % |
| key_pairs                         | 0          | 100        |           0 % |
| metadata_items                    | 0          | 128        |           0 % |
| ram                               | 512        | 51200      |           1 % |
| reservation_expire                | 0          | 86400      |           0 % |
| security_group_rules              | 0          | 20         |           0 % |
| security_groups                   | 0          | 10         |           0 % |
| volumes                           | 2          | 10         |          20 % |
+-----------------------------------+------------+------------+---------------+

The above was generated using a custom script which can be found on GitHub (https://github.com/cybera/novac/blob/dev/libexec/novac-quota-report).

Intelligent Alerting

Intelligent alerting can be thought of as a form of continuous integration for operations. For example, you can easily check to see if Glance is up and running by ensuring that the glance-api and glance-registry processes are running or by seeing if glace-api is responding on port 9292.

But how can you tell if images are being successfully uploaded to the Image Service? Maybe the disk that Image Service is storing the images on is full or the S3 back-end is down. You could naturally check this by doing a quick image upload:

#!/bin/bash 
# 
# assumes that reasonable credentials have been stored at
# /root/auth 
 
      
. /root/openrc 
wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img  
glance image-create --name='cirros image' --is-public=true --container-format=bare --disk-format=qcow2 < cirros-0.3.0-x8
6_64-disk.img

By taking this script and rolling it into an alert for your monitoring system (such as Nagios), you now have an automated way of ensuring image uploads to the Image Catalog are working.

Intelligent alerting takes a considerable more amount of time to plan and implement than the other alerts described in this chapter. A good outline to implement intelligent alerting is:

• Review common actions in your cloud

• Create ways to automatically test these actions

• Roll these tests into an alerting system

Some other examples for Intelligent Alerting include:

• Can instances launch and destroyed?

• Can users be created?

• Can objects be stored and deleted?

• Can volumes be created and destroyed?

Trending

Trending can give you great insight into how your cloud is performing day to day. For example, if a busy day was simply a rare occurrence or if you should start adding new compute nodes.

Trending takes a slightly different approach than alerting. While alerting is interested in a binary result (whether a check succeeds or fails), trending records the current state of something at a certain point in time. Once enough points in time have been recorded, you can see how the value has changed over time.

All of the alert types mentioned earlier can also be used for trend reporting. Some other trend examples include:

• The number of instances on each compute node

• The types of flavors in use

• The number of volumes in use

• The number of Object Storage requests each hour

• The number of nova-api requests each hour

• The I/O statistics of your storage services

As an example, recording nova-api usage can allow you to track the need to scale your cloud controller. By keeping an eye on nova-api requests, you can determine if you need to spawn more nova-api processes or go as far as introducing an entirely new server to run nova-api. To get an approximate count of the requests, look for standard INFO messages in /var/log/nova/nova-api.log:

# grep INFO /var/log/nova/nova-api.log | wc

You can obtain further statistics by looking for the number of successful requests:

# grep " 200 " /var/log/nova/nova-api.log | wc

By running this command periodically and keeping a record of the result, you can create a trending report over time that shows whether your nova-api usage is increasing, decreasing, or keeping steady.

A tool such as collectd can be used to store this information. While collectd is out of the scope of this book, a good starting point would be to use collectd to store the result as a COUNTER data type. More information can be found in collectd’s documentation (https://collectd.org/wiki/index.php/Data_source)