As you know, Nova works in concert with other OpenStack services, including Keystone, Glance, Neutron, and others. As a result, it is possible that problems with these services may manifest themselves as problems with Nova.
Nova makes use of a database to support its functionalities. If this database is not running or is improperly configured, it can cause your Nova installation to be unusable. To troubleshoot the Nova database, begin by making sure that the database engine is running. For example, when using MySQL, you can check the status by running the following command:
service mysql status
If the database is running, it will be indicated in the output of this command:
mysql start/running, process 15628
After confirming that the database engine is running, you should try to connect to the database server using the same parameters that are set up in the Nova configuration file. You can look up the connection information by running the following command:
less /etc/nova/nova.conf | grep "connection ="
The preceding command will search through the Nova configuration file, which is located at /etc/nova/nova.conf
. In this file, we are looking for a variable named connection
. This setting is typically under the stanza named [database]
. The value of the variable will follow a format similar to the following format if you are using MySQL:
mysql://nova:novapwd@mydbserver/nova
From the preceding connection string, we see that our database server is running on the host named mydbserver
with a database user named nova
, and a password novapwd
. We can use this information to attempt to connect to the database directly:
mysql -u nova -p -h mydbserver nova
On successful connection, a message similar to the one shown here will be printed on your console:
If the connection is unsuccessful, you will see a message like this one:
This error indicates that we need to troubleshoot further. There are several possible causes behind this error. A few potential reasons include the following:
To troubleshoot further, you will need root access to the database server. Attempt to connect to the database using your root user. Accomplishing this step will also allow you to confirm the value for the database host:
mysql -u root -p -h mydbserver
Once you have logged into the database server as a root user, you will want to check the permissions for the Nova database user. An example of how to do this is included here:
show grants for 'nova'@'locahost'
The output of the preceding command will be similar to what is printed in the following screenshot:
The nova
user needs to be granted all permissions on the Nova database. After confirming that the nova
user has the correct permissions, you want to make sure that the user's password is correct. You can change the nova
user's password by executing the following set of commands:
After executing this series of commands, the password for the nova
user will be updated. Make sure that this new value is properly reflected in the connection
variable under the [database]
stanza in the /etc/nova/nova.conf
file.
After you have confirmed that your nova
database user is set up correctly, you will want to make sure that the Nova database is properly initialized. The first step is to make sure that the Nova database exists. In MySQL, you can check this by running the following command:
show databases;
Make sure you see a database named nova
in the list:
If the Nova database does not exist, you need to create it. You can do this using the standard SQL syntax. Take the following line of code as an example:
mysql> CREATE DATABASE nova;
Once you have confirmed that the database exists, you need to make sure that the required tables are present in the database:
mysql> show tables;
After running the preceding command, you will see list of tables in the Nova database. The following output is a partial list of tables. There are around 100 tables in total.
When you run show tables
and the command comes back empty, you will need to initialize your Nova database:
nova-manage db sync
The preceding command does not return an output, but it will create the necessary tables in the Nova database. Confirm this by running the show tables command again, after which you should see that your database is populated.
The most obvious dependency Nova has is Keystone. Nova leverages Keystone to provide authentication services for all the Nova API calls. So, if Keystone is not running as expected, you will come to know this as soon as you attempt to use the OpenStack client:
openstack server list
If you try to use the preceding command to list the Nova servers while Keystone is not working, you will see an error like the one shown here:
We will begin troubleshooting the integration between Nova and Keystone by making sure that Keystone is up and running:
ps -aux | grep keystone
This command should return an output similar to what is shown in the following screenshot:
If you do not see any Keystone processes running, refer to Chapter 2, Troubleshooting OpenStack Identity, in this book for Keystone troubleshooting tips.
Next, we will make sure that the Nova service user is set up correctly. By convention, this service user will be named nova
. Confirm that this user exists by running the following command:
openstack user list
Running this command will return a list of users in Keystone. An example of the output is included in this screenshot:
Make sure that there is a user named nova
. Secondly, this user should belong to a project/tenant named service
, by convention. Make sure that the service project exists by running the following command:
openstack project list
This command will list the projects (tenants) configured in our Keystone installation:
In the output of this command, you should see a project named service
, as demonstrated in the preceding screenshot. Next, we need to confirm that the nova
user is assigned to the service project. We can accomplish this using the next command:
openstack user show nova
The preceding user show
command will display details for the nova
user as shown here:
In the project_id
field, we are looking for a project ID that matches the ID of our service
project displayed in the preceding OpenStack project-list command. The project ID we are looking for is 831669f19f3b4505a3b8b132e88a3b36
. The output tells us that our nova
user is indeed a part of the service
project.
Next, confirm that the nova
user has the correct roles in Keystone. The following command can help us with that:
openstack user role list --project service nova
This command will list the roles for the glance
user in the context of the service
project:
Confirm that the nova
user has the admin role in the service project.
Finally, confirm the password for the nova
user. Use the OpenStack command-line client to authenticate as the nova
user. Run a command similar to the one shown here:
openstack --os-username nova --os-password nova --os-project-name service project list
The preceding command is the Openstack project list command, with a few arguments passed in on the command line. The arguments passed in will override the values we have in our openrc
file. This allows us to test authentication with different users:
--os-username
: This indicates the username of the user we want to use. For this test, the username is nova
.--os-password
: This is the value we are targeting with this test. We want to make sure that the password is set and working as expected. For my test, I'm expecting the password to be nova
.--os-project-name
: Finally, we need to pass in the project name so that Keystone knows which project to authenticate under. Our user may have different roles under different projects. In our example, we are testing the nova
user under the service
project.If the password for the nova
user is equal to nova
, as I expect, this command will return a list of projects in keystone, as follows:
Suppose that I get an error like the one shown here:
The request you have made requires authentication. (HTTP 401) (Request-ID: req-707fd197-ae15-4698-ac04-1de4c92acfe9)
This error tells me that the password is not nova
, as I anticipated. At this point, I can try a few more password possibilities or I can reset the password. To reset the nova
user's password, you can use a command like the one shown here:
openstack user set --password-prompt nova
After running this command, you will be prompted to enter a new password for the nova
user. You will then be asked to type that password a second time to confirm it. After you reset the password, you can test it as we did earlier.
Finally, we need to confirm that our credentials are set properly in the Nova configuration files. In the /etc/nova/nova.conf
file, there should be a keystone_authtoken
stanza. Under this stanza, make sure that the values of the admin_user
and admin_password
fields match with the values you have set in Keystone. Consider the following example:
In the preceding example, admin_ user
is set to nova
and admin_password
is set to nova
. Replace these values with the username and password you have configured for your installation.
At this stage, we should confirm that the service catalog contains the correct endpoints for the Nova service with the following command:
openstack endpoint show nova
When you run the endpoint show
command, as demonstrated in the preceding code, you will see the details for the Nova endpoint in the service catalog. An example of a typical output is included in the following screenshot:
Confirm that the values of the adminurl
, internalurl,
and publicurl
fields are correct. If any of these values are incorrect, you have the following three options to correct the issue:
endpoint delete
command to remove the current endpoints, followed by the OpenStack endpoint create
command.Nova depends on Glance when retrieving images while launching instances. If there is an issue that you're facing with Glance, you will most likely see it quickly when you attempt to launch a new instance using the openstack server create
command:
openstack server create --flavor m1.tiny --image cirros-qcow2 server1
When this command is executed successfully, it will return a summary of the instance being created. If there is a problem with the Glance and Nova configurations, then you will see an error like the one shown here:
Could not find resource cirros-qcow2
If you run the same command with the --debug
flag, you will see a more detailed error message like the one shown in the following screenshot:
This error clearly points to the fact that glance-api
is not running or Glance is not configured correctly in the nova.conf
file. The Glance configuration for Nova can be found under the [glance]
stanza in /etc/nova/nova.conf
as shown here:
As demonstrated in the preceding code, you need to confirm the value of the host
parameter for Glance. Once you confirm the Glance configuration in nova.conf
, refer to Chapter 3, Troubleshooting the OpenStack Image Service, for more tips on Glance troubleshooting.
Nova also depends on Neutron to provide networking for the instances. Nova will make API calls to Neutron after authenticating via Keystone. When attempting to launch an instance, you may come across and error like the one shown in the following screenshot:
Upon further troubleshooting, you may see errors in nova-api.log
similar to the ones shown here:
If you run across a similar error, make sure that the Neutron server is running and configured correctly in the nova.conf
file.
The preceding screenshot shows an example of the Neutron configuration in the nova.conf
file. Make sure that this stanza exists, and the username and password values are correct for the Neutron service user. Once you have confirmed this, you can review the Chapter 4, Troubleshooting OpenStack Networking, for more tips.