Heat is the code name for the OpenStack orchestration service. Much like a conductor works with a collection of musicians and leads them to make beautiful music together, so does Heat work with a collection of OpenStack services in an effort to make cloud building easier. Heat provides a means by which users can coordinate services such as Nova, Neutron, Glance, Keystone, and others. Heat makes it easy to plug the OpenStack services together into Stack. When it comes to troubleshooting Heat, there are two stages you will need to be concerned with. The first is Heat itself and the second is all the OpenStack services Heat leverages to do its job. In this chapter, we will look at the following topics:
- Troubleshooting Heat services
- Fixing Heat authentication issues
- Correcting common Heat template errors
A properly running Heat installation will have at least two processes: the Heat API process (heat-api) and the Heat engine (heat-engine). In addition, you can also optionally run the Heat Cloud Formation compatibility API (heat-api-cfn). This API makes heat compatible with the API provided by the AWS Cloud Formation product. You can confirm that these processes are running by executing the following command:
ps –aux | grep heat
This command should return an output similar to the following output:
You can also leverage the pgrep command to check the Heat processes:
pgrep –l heat
The output from this command will be similar to the following output:
One way to confirm that the heat-api process is running as expected is to use the Heat command-line tool. For example, you can execute the stack-list command to check this:
heat stack-list
When the heat-api process runs as expected, this command returns a list of your current stacks or an empty list if you haven't created any stacks yet, as demonstrated in the following screenshot:
Suppose that you attempt to use the Heat command-line tool and you receive an error as follows:
This error message is a clear sign that your heat-api process is not running. You can also leverage the --debug argument on the heat command in order to see more detail of the error:
heat --debug stack-list
As with most of the other OpenStack command-line clients, the --debug argument is available as a tool to assist you in troubleshooting.
You can also use the Heat API to confirm the successful status of the heat-api process. If you run a basic query on the Heat API, it should return a list of the API versions supported by Heat. Take the following line of code as an example:
curl -i -H "Content-Type: application/json" http://127.0.0.1:8004/
The value returned from the preceding curl call should be similar to the following output:
When you run the preceding curl command and heat-api does not run, you will see an error similar to the one shown here:
To start heat-api, you can typically use one of the following commands:
start heat-api service heat-api start
Once you have attempted to start the service, use the ps or pgrep commands, as demonstrated earlier, to confirm that the service has started successfully. If the service does not appear to start as expected, you should attempt to start it manually. When starting the service manually, any error that occurs during the startup process will be printed directly to the console for further troubleshooting. To start heat-api manually, you will have to run the following command:
sudo -u heat heat-api --config-file=/etc/heat/heat.conf
Once you execute the preceding command, you should see startup log messages printed to the console. Check for any errors, failures, or trace messages. You may also want to execute heat commands to test whether heat-api is working, for example, the heat stack-list command we used earlier. If the heat-api service works as expected when you start it manually, but incorrectly when started with start heat-api or service heat-api start, then you will want to validate the init scripts for the heat service. Check your init script for the correct path to the configuration files and log file. Also, make sure that your Linux heat user has write permissions for the config directory and logging file, for example, /etc/init/heat-api.conf.
Similar to our approach of troubleshooting the heat-api process, we can confirm whether or not the heat-engine service is running as expected by executing a heat command:
heat stack-list
As we saw in the preceding heat-api section, this command should return a list of stacks or an empty list if your installation does not have any stacks. If the command times out with an error similar to the following one, this is a sign that there may be trouble with the heat-engine process:
You can further troubleshoot this type of error by looking at the heat logs. For example, the following trace is printed to /var/log/heat/heat.log in my installation. You can find these logs on the server running the heat-api process. In most deployments, this will be the controller node. The logging directory is set in the heat.conf file at /etc/heat/heat.conf under the attribute named log_dir.
The Heat command-line tool offers the service-list command, which can be useful when troubleshooting the heat-engine process. When heat-engine runs as expected, the output will be similar to the output shown here:
The heat service-list command will return a list of those Heat engines that are active in your installation. If you do not have any active Heat engines, the output returned will be similar to the output shown in this screenshot:
Each of the methods in this section will help you diagnose a problem with the heat-engine process. If you determine that the process is not running, it can be started with one of the following commands:
start heat-engine service heat-engine start
Once you have started the heat-engine process, you can confirm that it is running successfully using the ps or pgrep command, as demonstrated earlier. If the process does not start as expected, you can attempt to run the process manually. When you start the process manually, the startup output will be printed to the terminal and may prove useful for further troubleshooting. To start the heat manually, you will need to run the following command:
sudo -u heat heat-engine --config-file=/etc/heat/heat.conf
Check the output of the preceding command for any ERRORS, WARNINGS, or failure lines. If heat-engine starts successfully when it's run manually, but does not start as expected when you use the start or service commands, you will want to check the heat-engine init scripts. Check your init script for the correct path to the configuration files and log file. Also make sure that your Linux heat user has write permissions for the config directory and logging file.