CHAPTER 13

Troubleshooting and Debugging

Chef Troubleshooting and Debugging

After having gone through the installation, configuration, and development aspects of chef, let’s now look at how we can troubleshoot and debug if something doesn’t work in the chef environment. In this chapter, we look at common themes in debugging and troubleshooting a chef environment.

Debugging Chef Client Run

Several approaches can be used to debug a chef client run.

Running Chef Client with an Empty Run List

At times, you may have issues with a chef client run; it may not run a recipe or may not behave as it should for chef runs.

Running the chef client with an empty run list will tell us whether the chef client failed because of recipes in the run list of the node, the chef client configuration of the node, or the issues with connectivity between the chef server and the client.

If chef client fails with an empty run list, it can be for one of the following reasons:

1. Ports required for a chef client are blocked by a firewall or the network connectivity between the server and the client is unavailable. Running an empty run list will indicate whether the connection has become timed out or there is a network down error (see Figure 13-1).

   

   Figure 13-1. Chef client run

2. The user does not have permissions to run chef client. When users do not have the required permissions on the chef client, they get a permission denied error (see Figure 13-2).

   

   Figure 13-2. Permissions error

3. There is an invalid validation key or invalid client.rb file to access the chef server. In the case of an invalid client.rb file or invalid validation key, you will see the connection refused error (see Figure 13-3).

   

   Figure 13-3. Configuration file missing

4. For a hosted chef, if the client node is behind proxy, invalid proxy settings can lead to failed chef client runs (see Figure 13-4). Due consideration should be given to correct proxy configuration to ensure that it allows connection between chef client and chef server.

Running Chef Client in Debug Mode

Running chef client in debug mode (see Figure 13-5) will give you the verbose output of each and every action executed by the chef client. You can specify the log level in the client.rb file get the output in a client.log file or you can run the debug level command directly from the terminal.

There are three main levels for debugging chef client runs.

1. Info: To run chef client in this mode run chef-client –l info.
2. Debug: To run chef client in this mode run chef-client –l debug.
3. Warn: To run chef client in this mode run chef-client –l warn.

Using Chef Client Log Files

The log file location has to be mentioned in the client.rb file present inside /etc/chef folder. Use the following parameter in the client.rb file to set the log details.

log_location "/var/log/chef/chef-client.log

Once the chef client run is finished, you can get insight about the chef client run from the log file (see Figure 13-6).

Types of Log Errors

A log file starts with the internal chef logs. Following are the types of warning and errors you find in the log file:

• WARN: You get a warning when you have situations like an empty run list.
• FATAL: This indicates that the chef resource has failed to execute. You can find what failed using the ERROR level in the log file.
• ERROR: This indicates that the chef client was unable to load the exception handlers, when it fails to execute a resource, or when a package installation fails.

Using Chef Handler Cookbook

Chef handler is a cookbook that helps in handling exceptions happening during a chef client run. This cookbook can be obtained from the chef community cookbooks. This cookbook has a chef_handler lightweight resource provider. This cookbook can be used to make product specific handlers, so that the chef client will handle the exceptions in the way that you specify in your cookbooks with chef_handler resource.

Put the recipe chef_handler at the start of the node’s run list to make sure that custom handlers are loaded early on in the chef run and available for the other recipes.

Download the chef_handler cookbook using knife (see Figure 13-7).

Knife cookbook download chef_handler

Untar the cookbook and Upload the cookbook to the chef server.

Knife cookbook upload chef_handler

Add the chef_handler recipe to the nodes run list (see Figure 13-8).

You can include the chef_handler libraries in your recipes by including the chef_handler recipe to your custom recipe. The chef_handler dependency has to be added to your cookbooks metadata.

include_recipe 'chef_handler'

Debugging Recipes Using Logs

Chef logs can be used to debug recipes. During a chef client run, the logs are written to the log file specified in the client.rb file. For debugging information, chef client has to run in debug mode. This can be set in the client.rb file using log_level parameter value to :debug.

Common Errors

Cookbook Not Found

We get this error if a cookbook is added to the run list of a node but the cookbook is not present on the server. In this case the chef client will fail and “cookbook does not exist on the server” will appear as shown in Figure 13-9.

This error can be resolved by adding the cookbook with a relevant name (see Figures 13-10 and 13-11) to the run list of the node.

Package Installation Error

If the package installation has any errors (see Figure 13-12), you can find the error in the log file with the package name associated with it.

This error can occur for any of the following reasons:

• Repository not configured.
• Package with that name does not exist in the repository.

This error can be resolved by configuring the repository and ensuring it is reachable. Also, you must provide a valid name for the package.

Using a Log Resource

A log resource can be used inside recipes to write to log files. For example, if you want to log an entry when a particular data bag item is used in your recipe, you can use the log resource in the data bag block to do that. Log resource uses Chef::Provider::Log::ChefLog during the chef client run. The screenshot in Figure 13-13 shows how to use the breakpoints in recipes.

Debugging Recipes Using Chef Shell

Chef shell is a recipe debugging tool which runs as an interactive ruby session. Using chef shell, you can run a recipe on debugging mode to trace the errors.

Configuring Chef Shell

The chef shell has three modes:

1. Stand-alone: it is the default mode and no cookbooks will be loaded in this mode. Chef shell stand-alone mode (see Figure 13-14) can be started using chef shell command.

   

   Figure 13-14. Stand-alone mode of chef shell

2. Chef solo: in this mode, the chef shell will have the chef solo functionalities (see Figure 13-15). In this mode, chef shell will load the chef solo cookbooks and JSON (Java Script Object Notations) attributes. You can activate the chef solo mode using “chef shell –s” command.

   

   Figure 13-15. Chef solo mode of chef shell

3. Chef client: in this mode, the chef shell will have the chef client functionalities (see Figure 13-16). To use this mode you have to place a chef-shell.rb file with few parameters inside the /etc/chef folder.

You can run the recipes from the nodes run list in debugging mode and you can trace errors in the recipe (see Figure 13-17). You can activate the chef client mode using “chef shell –z” command.

Whenever the chef shell is loaded in chef client mode, all the recipes in the nodes run list will be loaded to the cache. You can debug the recipes in the run list using run chef command (see Figure 13-18).

Debugging Recipes Using Breakpoint Resource

You can do block level debugging using breakpoint resource (see Figure 13-19). While testing cookbooks include the breakpoint resource after every resource block to have a block level debugging.

To debug a recipe with breakpoint, change the chef shell mode to recipe mode (see Figure 13-20) using “recipe_mode” command and run the chef client using the “run_chef” command.

The “run_chef” command will run the recipe until the breakpoint specified in the recipe. Once you have debugged until the breakpoint, run the “chef_run.resume” command to resume the chef client run from the breakpoint (see Figure 13-21).

Troubleshooting Chef Client

Chef Client Fails to Run a Recipe Successfully at Bootstrapping

Run chef client to set up the connection with the chef server. It will run with an empty run list. After it is connected, log in to http://yourcherver.com:443, add the roles/recipes to the client node, and run the chef client from the client again.

Reregistering a Removed Client

The reregistration process is required if the client is unable to authenticate itself with the chef server. To reregister a removed client from the chef server, remove the client.pem file from /etc/chef folder and run the chef client on the node. Before running the chef client, make sure that the node object of the removed client is also removed from the chef server.

Issues Registering Chef Client with the Server

While registering a node with chef server, the registration might fail for the following reasons:

1. The client name already exists in the chef server (see Figure 13-22): this issue can be fixed by deleting the existing client name from the server.

   

   Figure 13-22. The client name that exists in error

2. Invalid chef server URL (uniform resource locator) (see Figure 13-23): check whether the chef server URL is specified correctly in the client.rb file of the node.

   

   Figure 13-23. The URL not valid error

3. Invalid validation key (Figure 13-24): check whether the validation key specified in the client.rb file and the validation key that is present inside the /etc/chef folder are valid.

401 Unauthorized Errors

401 Unauthorized errors happen (see Figure 13-25) for the following reasons:

1. Incorrect client.pem file: this error can be rectified by deleting the existing client.pem file from the /etc/chef folder and reregistering the node with the chef server.

Removing chef node from a server:

knife client delete NODENAME
knife node delete NODENAME

On an affected node, remove the client.pem file and run the chef client again.

sudo rm /etc/chef/client.pem
sudo chef-client

Clock Synchronization Error

This error happens when your client node’s clock drifts from the actual time by more than 15 minutes (see Figure 13-26). You can fix this by syncing your clock with an NTP (Network Time Protocol) server. Run the chef client after synching the clock.

No Such File or Directory: /etc/chef/validation.pem

This error happens when the validation key is not present in the default /etc/chef folder. This can be rectified by copying the validator key from the chef server to the nodes /etc/chef folder.

Cannot Find Config File

This error happens when the client.rb file is not present in the default chef directory on the node (see Figure 13-27). You can work around this issue by adding the path to of client.rb file using –c switch.

chef-client -c C:chefclient.rb