CHAPTER 12

Cloud Provisioning Using Chef

Provisioning Using Vagrant and Chef

Until now we have seen how to leverage chef to provision and configure application and database environments. In this chapter we introduce our readers to how Vagrant and chef can work together to bring up complete environments including the virtual infrastructure.

Vagrant is an open source infrastructure provisioning solution that integrates with VMware, AWS, VirtualBox, and other hypervisor and cloud environments to provision virtual and cloud infrastructure.

After provisioning of the cloud or virtual infrastructure through Vagrant, chef can be used to deploy and configure the application servers and databases on the virtual machine.

Thus a combination of Vagrant and chef can provide immense benefits to enterprises looking to automate the infrastructure and environment and build use cases.

Just as chef has a concept of recipe, Vagrant has the concept of a Vagrantfile which provides the configuration details for the virtual machine or cloud infrastructure.

A much used use case of the Vagrant tool is to create disposable infrastructure and environments for developers and testers. As development and test environments need frequent provisioning and deprovisioning, the power of Vagrant to quickly assemble the required infrastructure helps the infrastructure and automation teams in fulfilling the demands of the development and test teams.

With its integrations to various hypervisors and cloud providers the same Vagrant tool can be used to provision the infrastructure in premise virtual infrastructure and the cloud.

Technically, Vagrant is a Ruby-based application. Vagrant can be easily deployed as a virtual machine on VirtualBox or VMware workstation.

Vagrant can quickly deploy the environment using a single command “vagrantup” once the Vagrant tool is up and running and is configured with correct Vagrantfile.

Providers and Provisioners

The development environment can be configured using Vagrant’s providers and provisioners.

Providers are the virtual machine solutions such as VirtualBox, VMWare, Amazon AWS, and Digital Ocean.

Provisioners are used to manage the configuration in your development environment. Chef and puppet are few examples of provisioners used by Vagrant. Using provisioners you can automate the application configuration process on a Vagrant box.

Installing Vagrant

Vagrant works on Windows (32 bit and 64 bit), Linux Debian (32 bit and 64 bit), Linux rpm based (32 bit and 64 bit), and MAC OS X (32 bit and 64 bit) environments.

Install Virtual Box

Vagrant works with virtual box in the back end. So, before installing Vagrant, virtual box has to be installed in your system. Virtual box has to be installed on a workstation and not on any virtual machine.

Go to www.virtualbox.org/wiki/Downloads. Download and install the latest virtual box software.

Install Vagrant on Windows

1. Go to www.vagrantup.com/downloads.
2. Download the Vagrant Windows MSI installer package (see Figure 12-1).

   

   Figure 12-1. Downloading Vagrant

3. Run the installer (see Figure 12-2).

   

   Figure 12-2. Running the Vagrant Installer

4. Click next.

   

   Figure 12-3. Installing Vagrant

5. Accept the license and click next.

   

   Figure 12-4. Accepting the Terms

6. Enter the destination folder for Vagrant and click next.

   

   Figure 12-5. Selecting a Installation Directory

7. Click install.

   

   Figure 12-6. Installing Vagrant

8. Click finish.

   

   Figure 12-7. Finishing installation

Verifying Vagrant Installation

Open the windows command prompt and check the Vagrant version to make sure it is installed in your system (see Figure 12-8).

1. vagrant –v

Configuring Vagrant

Vagrant Boxes (OS Images)

Vagrant uses the notion of “boxes” to describe preconfigured base virtual machines from which we can work. Creating a virtual machine from scratch is a tedious and time-consuming process. Vagrant, on the other hand, creates a virtual machine quickly by cloning the preconfigured virtual machines. These preconfigured base images are termed as boxes in Vagrant.

You can download Vagrant boxes (Linux based) based on your requirement from here www.vagrantbox.es/.

Two Methods to Download Vagrant Box

1. Download and add the Vagrant base box (see Figure 12-9) with preconfigured chef using the following command:

   #syntax vagrant box add {boxname} {box provider url}
   vagrant box add ubunt12.04http://files.vagrantup.com/precise32.box

   

   Figure 12-9. Adding Vagrant box directly

   Vagrant boxes will be downloaded to the users/.vagrant.d/boxes folder.

2. Download the box directly from https://files.vagrantup.com to your file system and you can add that box to Vagrant by specifying the local file system path (see Figure 12-10).

vagrant box add vagrant_demo C:UsersAdministratorcentosCentOS-6.4-i386-v.box

Vagrantfile

The next step is to configure Vagrant using a Vagrantfile.

1. The first step is to create a root directory. A few configurations related to Vagrant will be relative to this directory
2. Mention the kind of box and provisioners you need to have in your environment in the Vagrantfile.
3. Vagrantfile can be created using vagrant-init command. This command will create a Vagrantfile in your current directory. Adding the existing box title to the vagrant-init command will create a Vagrantfile with the configuration details of the mentioned box.

#syntax vagrant init {box-title}
 
vagrant init ubunt12.04

The foregoing command creates a Vagrantfile (see Figure 12-11) based on the base box titled ubunt12.04.

Provisioning a New Instance

You can start the virtual machine (see Figure 12-12) using “vagrant up” command. When you use this command, make sure that your present working directory is the directory which contains the Vagrantfile.

Here we are using centos box which has already been downloaded using “vagrant box add” command. The Vagrantfiles relative to centos box are present inside the c:/users/centos folder.

Connecting to a Virtual Machine

The virtual machine (VM) which got spun up by Vagrant can be connected using the “vagrant ssh” command. This command will drop you into a full-fledged SSH session. By default, a Vagrant VM will have username and password “vagrant.”

The network details are fetched from the Vagrantfile. You can have a host-only access static IP or dynamic IP allocated by the public network.

Also, you can spin up multiple VMs using one Vagrantfile by defining more than one VM in the Vagrantfile. By default there will be only a box definition inside a Vagrantfile (e.g., config.vm.box = "vagrantdemo"). SSH access can be made to individual VMs using the name given in the definition.

When multiple VMs are launched using a single Vagrantfile, the SSH command should include the VM name to connect to it (see Figure 12-13).

Suspending the Virtual Machine

By suspending the VM, you can get a point in time state of the virtual machine (see Figure 12-14). The Vagrant “suspend” command will not shut down the machine completely; you can resume the VM anytime.

Halting the Virtual Machine

“Vagrant halt” command shuts down the VM (see Figure 12-15). You can provide –f switch to forcefully shut down the machine. You have to execute this command from the root folder of the VM.

Destroying the Virtual Machine

“Vagrant destroy” command will stop the VM and destroys all the resources associated with it (see Figure 12-16). Once you issue the destroy command, it will ask for confirmation to destroy the VM.

Installing Vagrant Plug-Ins

The following command can be executed from anywhere. Plug-ins are necessary to integrate Vagrant with other services like AWS, Azure, Hyper-V, and so on.

Command: vagrant plugin install <name>

Vagrant plug-ins will be installed from RubyGems and this command also updates the existing gem. The command “vagrant plugin update” is specifically used to update the plug-in. You can also install multiple plug-ins by providing multiple names with the command.

Vagrant Provisioning Using Chef

Provisioners in Vagrant let you automatically install and configure software on the Vagrant machine as part of the “vagrantup” process. If you want to just use Vagrant SSH and install the software by hand, that will work. By using provisioners you can automate the repeatable tasks. To use provisioners, make sure that the particular provisioner is included in the Vagrant box you download. For example, if you want to use chef provisioner, use a box that contains chef client.

The configuration for provisioning by default will be commented out in the Vagrantfile. We have to work on this file and fill in the required details for chef to work with Vagrant. The file will look as shown in Figures 12-17 and 12-18. This file will be present in the root folder you created for the VM.

Chef Solo Provisioner

Using chef solo provisioner you can provision a guest OS using chef solo. All the configurations needed for chef solo have to be mentioned in the Vagrantfile.

Options

This section lists the complete set of available options for the chef solo provisioner. More detailed examples of how to use the provisioner are available here.

Note that only the chef solo specific options are shown in the following list. There is also a large set of common options available with both the chef solo and chef client provisioners.

cookbooks_path (string or array): A list of paths to where cookbooks are stored. By default this is “cookbooks,” expecting a cookbooks folder relative to the Vagrantfile location.

data_bags_path (string): A path where data bags are stored. By default, no data bag path is set.

environments_path (string): A path where environment definitions are located. By default, no environments folder is set.

environment (string): The environment you want the chef run to be a part of. This requires Chef 11.6.0 or later, and it requires the environments_path to be set.

recipe_url (string): URL to an archive of cookbooks that chef will download and use.

roles_path (string or array): A list of paths where roles are defined. By default, it is empty. Multiple role directories are only supported by Chef 11.8.0 and later.

Specifying a Run List

The run list specified in the Vagrantfile will be configured when the “vagrantup” command is issued.

Vagrant.configure("2") do |config|
  config.vm.provision "chef_solo" do |chef|
    chef.add_recipe "httpd"
  end
end

This will make Vagrant run chef solo with the “httpd” cookbook. The cookbook has to be present in the cookbooks directory of your VM’s root directory. The directory structure should look as shown in Figure 12-19.

Specifying Roles

You can specify chef roles in a Vagrantfile. By default, Vagrant will look for the roles directory in the VM’s root folder. You can specify the path if the roles folder is in some other location.

Vagrant.configure("2") do |config|
  config.vm.provision "chef_solo" do |chef|
    chef.roles_path = "vagrant_roles"
    chef.add_role("webserver")
  end
end

Using Data Bags

You can use data bags with chef solo provisioner. Data bags can be used inside recipes that chef solo uses. By default, Vagrant will search for the data bags folder in the VM root directory.

Vagrant.configure("2") do |config|
  config.vm.provision "chef_solo" do |chef|
    chef.data_bags_path = "data_bags"
  end
end

Specifying Node Name

You can set a custom node name by mentioning the node_name option.

Vagrant.configure("2") do |config|
  config.vm.provision "chef_solo" do |chef|
    chef.node_name = "node_name"
  end
end

Custom JSON Data

Additional configuration data for chef attributes can be passed in to chef solo.

You can also pass parameters using JSON (JavaScript Object Notation) in the chef solo provisioner.

Vagrant.configure("2") do |config|
  config.vm.provision "chef_solo" do |chef|
      chef.json = {
      "httpd" => {
        "listen_address" => "10.0.1.5"
      }
    }
  end
end

Provisioning Chef Client

Using chef client provisioner you can provision a Vagrant guest VM which will be registered to the existing chef server. You have to mention the validation key and chef server URL in the Vagrantfile. By default the chef client will look for the validation key in the VM root directory. If it is not present in the root directory, you have to provide the path of the validation key in the validation_key_path parameter.

Vagrant.configure("2") do |config|
  config.vm.provision "chef_client" do |chef|
    chef.chef_server_url = "https://my_chef_server.com:443/"
    chef.validation_key_path = "validation_key.pem"
  end
end

Adding Run List

You can override the run list specified by chef server by providing run list in the Vagrantfile. The run lists and recipes added to the Vagrantfile will be pulled from the chef server and applied to the Vagrant VM.

Vagrant.configure("2") do |config|
  config.vm.provision "chef_client" do |chef|
    # Adding a recipe
    chef.add_recipe "apache"
    # adding a role
    chef.add_role "web"
  end
end

Deleting the Node Details

Once a client is registered with the chef server, two entries are registered in the chef server, a node object and a client entry. These entries have to be deleted for Vagrant VM which is to be destroyed. It can be done by setting the values of two parameters to true.

chef.delete_node = true
chef.delete_client = true

Verifying Chef Client Registration with Chef Server

1. Run chef client command from the vagrant box by connecting using Vagrant SSH command to ensure that your vagrant box has been successfully registered with the chef server (see Figure 12-20).

AWS and Chef Provisioning Using Vagrant

Installing the Plug-ins

The following plug-ins have to be installed in Vagrant for instance and chef client provisioning on AWS (see Figure 12-21):

1. vagrant-aws

   

   Figure 12-21. Installing Vagrant AWS plug-in

2. vagrant-omnibus

   This plug-in is used to install chef client on the target AWS instance using the chef omnibus installer (see Figure 12-22).

   

   Figure 12-22. Installing vagrant-omnibus plug-in

Adding the AWS Box to Vagrant

Install a Vagrant box set with an AWS provider (see Figure 12-23) using the following URL:

https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

Vagrantfile for AWS Provisioner

The Vagrantfile for AWS provisioning should mention the following configurations in the screenshot (see Figure 12-24). By default, it launches instance to EC2-classic, and if EC2-classic is not available in that region, it will be launched in the default VPC (virtual private cloud). For deploying instances in VPC, the following three attributes have to be mentioned in the Vagrantfile:

aws.private_ip_address = "10.10.10.10"
aws.security_groups = ["sg-123z212d"]
aws.subnet_id = "subnet-23b9e7h1"

Save the Vagrantfile and use the “vagrantup --provider=aws” command to provision AWS instance with chef client (see Figures 12-25 and 12-26). You have to execute this command from the root folder.

You will be able to see the machine being provisioned in the AWS console (see Figure 12-27).

Log in to your instance and run chef client. It should be able to interact with the chef server as shown in Figure 12-28.

Provisioning Using Knife

Knife EC2 Plug-ins

The knife EC2 plug-in is used to manage or create instances on AWS. In the upcoming section we cover the installation, configuration, use of the plug-in.

Installing the Plug-in

Chef expects the knife plug-in to be located in the /opt/chef/embedded/bin directory. Any plug-in for knife can be installed using the following syntax:

/opt/chef/embedded/bin/gem install plugin_name

To install the knife EC2 plug-in (see Figure 12-29) run the following command:

/opt/chef/embedded/bin/gem install knife-ec2

Configuring

After you complete the installation, the next step is to configure the plug-in so that your plug-in is able to communicate with your AWS account (see Figure 12-30).

Enter the access key and secret key for your account using the following syntax in the knife configuration file:

Knife[:aws_access_key_id] = "Your_Access_Key"

Knife[:aws_secret_access_key] = "Your_Secret_key"

Once you complete the configuration, we are ready to use the plug-in. See Figure 12-31 for the available commands in the EC2 plug-in.

To view the list of instances in your account, run the following command.

Knife ec2 server list

This command would list out the servers in us-east1 region by default. If we want to list out the servers in any other region, we can do it using the –region option (see Figure 12-32).

To view the type of instances available for our account, run the following command:

Knife ec2 flavor list

This would list out the type of instances available in your account (see Figure 12-33).

The next step is to provision the server on EC2 (see Figure 12-34). We will use the knife EC2 server to create a command for the same.

Some of the important options needed in order to provision any instance are

• -f or -flavor: The type of instance to provision.
• -region: The region in which our instance will be provisioned.
• -I or –image: The image to use while provisioning the instance.
• -G or –groups: The name of the security group that would be attached to our instance. If we want to attach more than one security group we can provide them in a command-separated format. These groups should be present in our account.
• -S or –ssh-key – The key that would be used to log in to our instance. This key should be created before running the command.
• -x: This option is used to specify the username that would be used to log in to the instance.

After the instance is provisioned, the details would be displayed on the console (see Figure 12-35).

To verify that your provisioning has been completed, go to the AWS console (see Figure 12-36).

Knife Azure Plug-ins

Chef expects the knife plug-in to be located in the /opt/chef/embedded/bin directory. Any plug-in for knife can be installed using the following syntax:

/opt/chef/embedded/bin/gem install plugin_name

To install the knife Azure plug-in (see Figure 12-37), run the following command:

/opt/chef/embedded/bin/gem install knife-azure

Configuring

After you have completed the installation, the next step is to configure the plug-in (see Figure 12-38) so that your plug-in is able to communicate with your Azure account.

Enter the access key and secret key for your account using the following syntax in the knife configuration file:

Knife[:azure_publish_setting_file] = "/path_to_file"

Once you have completed the configuration, we are ready to use the plug-in. Figure 12-39 shows the commands available in the Azure plug-in.

To view the list of images available in your account, run the following command:

Knife azure image list

This command would list out the images available in all the regions (see Figure 12-40).

The next step is to provision the server on Azure. We will use the knife Azure server to create a command for the same.

Some of the important options needed in order to provision any instance (see Figure 12-41) are

• -azure-dns-name: The DNS (Domain Name System)that would be given to our instance
• -azure-source-image: The image that would be used to provision the instance.
• -I or –image: The image to use while provisioning the instance.
• –winrm-password: The password that would be used to log in to the instance.
• -winrm-user: This option is used to specify the username that would be used to log in to the instance.
• -u: This option is used to specify the UDP (User Datagram Protocol) ports that would be opened on the instance.
• -Z: This option is used to specify the size of the VM.

After the instance is provisioned, the details would be displayed on the console.

To verify that your provisioning has been completed (see Figure 12-42), go to the Azure console.