Troubleshooting and Debugging
Chef Troubleshooting and Debugging
After having gone through the installation, configuration, and development aspects of chef, let’s now look at how we can troubleshoot and debug if something doesn’t work in the chef environment. In this chapter, we look at common themes in debugging and troubleshooting a chef environment.
Debugging Chef Client Run
Several approaches can be used to debug a chef client run.
Running Chef Client with an Empty Run List
At times, you may have issues with a chef client run; it may not run a recipe or may not behave as it should for chef runs.
Running the chef client with an empty run list will tell us whether the chef client failed because of recipes in the run list of the node, the chef client configuration of the node, or the issues with connectivity between the chef server and the client.
If chef client fails with an empty run list, it can be for one of the following reasons:
Figure 13-1. Chef client run
Figure 13-2. Permissions error
Figure 13-3. Configuration file missing
Figure 13-4. Proxy setting error
Running Chef Client in Debug Mode
Running chef client in debug mode (see Figure 13-5) will give you the verbose output of each and every action executed by the chef client. You can specify the log level in the client.rb file get the output in a client.log file or you can run the debug level command directly from the terminal.
Figure 13-5. Debugging chef client
There are three main levels for debugging chef client runs.
Using Chef Client Log Files
The log file location has to be mentioned in the client.rb file present inside /etc/chef folder. Use the following parameter in the client.rb file to set the log details.
log_location "/var/log/chef/chef-client.log
Once the chef client run is finished, you can get insight about the chef client run from the log file (see Figure 13-6).
Figure 13-6. Using log files
Types of Log Errors
A log file starts with the internal chef logs. Following are the types of warning and errors you find in the log file:
Using Chef Handler Cookbook
Chef handler is a cookbook that helps in handling exceptions happening during a chef client run. This cookbook can be obtained from the chef community cookbooks. This cookbook has a chef_handler lightweight resource provider. This cookbook can be used to make product specific handlers, so that the chef client will handle the exceptions in the way that you specify in your cookbooks with chef_handler resource.
Put the recipe chef_handler at the start of the node’s run list to make sure that custom handlers are loaded early on in the chef run and available for the other recipes.
Download the chef_handler cookbook using knife (see Figure 13-7).
Knife cookbook download chef_handler
Figure 13-7. Downloading chef handler cookbook
Untar the cookbook and Upload the cookbook to the chef server.
Knife cookbook upload chef_handler
Add the chef_handler recipe to the nodes run list (see Figure 13-8).
Figure 13-8. Adding the cookbook to the run list
You can include the chef_handler libraries in your recipes by including the chef_handler recipe to your custom recipe. The chef_handler dependency has to be added to your cookbooks metadata.
include_recipe 'chef_handler'
Debugging Recipes Using Logs
Chef logs can be used to debug recipes. During a chef client run, the logs are written to the log file specified in the client.rb file. For debugging information, chef client has to run in debug mode. This can be set in the client.rb file using log_level parameter value to :debug.
Common Errors
Cookbook Not Found
We get this error if a cookbook is added to the run list of a node but the cookbook is not present on the server. In this case the chef client will fail and “cookbook does not exist on the server” will appear as shown in Figure 13-9.
Figure 13-9. Cookbook not found
This error can be resolved by adding the cookbook with a relevant name (see Figures 13-10 and 13-11) to the run list of the node.
Figure 13-10. Adding the cookbook with the right name
Figure 13-11. Error resolved
Package Installation Error
If the package installation has any errors (see Figure 13-12), you can find the error in the log file with the package name associated with it.
Figure 13-12. Unable to install package
This error can occur for any of the following reasons:
This error can be resolved by configuring the repository and ensuring it is reachable. Also, you must provide a valid name for the package.
Using a Log Resource
A log resource can be used inside recipes to write to log files. For example, if you want to log an entry when a particular data bag item is used in your recipe, you can use the log resource in the data bag block to do that. Log resource uses Chef::Provider::Log::ChefLog during the chef client run. The screenshot in Figure 13-13 shows how to use the breakpoints in recipes.
Figure 13-13. Using the log resource
Debugging Recipes Using Chef Shell
Chef shell is a recipe debugging tool which runs as an interactive ruby session. Using chef shell, you can run a recipe on debugging mode to trace the errors.
Configuring Chef Shell
The chef shell has three modes:
Figure 13-14. Stand-alone mode of chef shell
Figure 13-15. Chef solo mode of chef shell
Figure 13-16. Chef client mode of chef shell
You can run the recipes from the nodes run list in debugging mode and you can trace errors in the recipe (see Figure 13-17). You can activate the chef client mode using “chef shell –z” command.
Figure 13-17. Running recipes
Whenever the chef shell is loaded in chef client mode, all the recipes in the nodes run list will be loaded to the cache. You can debug the recipes in the run list using run chef command (see Figure 13-18).
Figure 13-18. Debugging recipes
Debugging Recipes Using Breakpoint Resource
You can do block level debugging using breakpoint resource (see Figure 13-19). While testing cookbooks include the breakpoint resource after every resource block to have a block level debugging.
Figure 13-19. Debugging recipes using breakpoint resource
To debug a recipe with breakpoint, change the chef shell mode to recipe mode (see Figure 13-20) using “recipe_mode” command and run the chef client using the “run_chef” command.
Figure 13-20. Debugging recipes using breakpoint resource with chef shell
The “run_chef” command will run the recipe until the breakpoint specified in the recipe. Once you have debugged until the breakpoint, run the “chef_run.resume” command to resume the chef client run from the breakpoint (see Figure 13-21).
Figure 13-21. Resuming the chef client
Troubleshooting Chef Client
Chef Client Fails to Run a Recipe Successfully at Bootstrapping
Run chef client to set up the connection with the chef server. It will run with an empty run list. After it is connected, log in to http://yourcherver.com:443, add the roles/recipes to the client node, and run the chef client from the client again.
Reregistering a Removed Client
The reregistration process is required if the client is unable to authenticate itself with the chef server. To reregister a removed client from the chef server, remove the client.pem file from /etc/chef folder and run the chef client on the node. Before running the chef client, make sure that the node object of the removed client is also removed from the chef server.
Issues Registering Chef Client with the Server
While registering a node with chef server, the registration might fail for the following reasons:
Figure 13-22. The client name that exists in error
Figure 13-23. The URL not valid error
Figure 13-24. The invalid validation key
401 Unauthorized Errors
401 Unauthorized errors happen (see Figure 13-25) for the following reasons:
Figure 13-25. Authentication error
Removing chef node from a server:
knife client delete NODENAME
knife node delete NODENAME
On an affected node, remove the client.pem file and run the chef client again.
sudo rm /etc/chef/client.pem
sudo chef-client
Clock Synchronization Error
This error happens when your client node’s clock drifts from the actual time by more than 15 minutes (see Figure 13-26). You can fix this by syncing your clock with an NTP (Network Time Protocol) server. Run the chef client after synching the clock.
Figure 13-26. A clock sync error
No Such File or Directory: /etc/chef/validation.pem
This error happens when the validation key is not present in the default /etc/chef folder. This can be rectified by copying the validator key from the chef server to the nodes /etc/chef folder.
Cannot Find Config File
This error happens when the client.rb file is not present in the default chef directory on the node (see Figure 13-27). You can work around this issue by adding the path to of client.rb file using –c switch.
chef-client -c C:chefclient.rb
Figure 13-27. Not able to find the client configuration file