External Facts

Would you like to provide facts without writing Ruby? This is now possible with external facts. There are two ways to provide external facts:

• Write fact data out in YAML, JSON, or text format.
• Provide a program or script to output fact names and values.

The program or script can be written in Bash, Python, Java, whatever... it’s only necessary that it can be executed by the Puppet agent. Let’s go over how to use these.

# three_simple_facts.yaml
---
my_fact: 'myvalue'
my_effort: 'easy'
is_dynamic: 'false'

# three_simple_facts.txt
my_fact=myvalue
my_effort=easy
is_dyanamic=false

#!/bin/bash
echo "my_fact=myvalue"
echo "my_effort=easy"
echo "is_dynamic=false"

$ $EDITOR facts.d/three_simple_facts.sh
$ chmod 0755 facts.d/three_simple_facts.sh
$ facts.d/three_simple_facts.sh
my_fact=myvalue
my_effort=easy
is_dynamic=false

Write-Host "my_fact=myvalue"
Write-Host "my_effort=easy"
Write-Host "is_dyanamic=false"

Custom (Ruby) Facts

To create new facts in Ruby for Facter to use, simply create the following directory in your module:

$ mkdir -p lib/facter

Ruby programs in this directory will be synced to the node at the start of the Puppet run. The node will evaluate these facts, and provide them for use in building the catalog.

Within this directory, create a Ruby script ending in .rb. The script can contain any normal Ruby code, but the code that defines a custom fact is implemented by two calls:

• A Ruby code block starting with Facter.add('fact_name')
• Inside the Facter code block, a setcode block that returns the fact’s value

For an easy example, let’s assume that hosts in a cluster have the same name. Unique numbers are added to each node to keep them distinct. This results in a common pattern with hostnames like:

• webserver01
• webserver02
• webserver03
• mailserver01
• mailserver02

You’d like to define Hiera data based on the cluster name. Right away, you might think to use the hostname command to acquire the node name. Facter provides a helper function to execute shell commands: Facter::Core::Execution.exec().

The code to derive a node’s cluster name from the hostname command could look like this:

# cluster_name.rb
Facter.add('cluster_name') do
  setcode do
    Facter::Core::Execution.exec("/bin/hostname -s | /usr/bin/sed -e 's/d//g'")
  end
end

This new fact will be available as $facts['cluster_name'] for use in manifests and templates.

It is best to run the smallest command possible to acquire a value, and then manipulate the value using native Ruby code. Here’s an example using Ruby native libraries to acquire the hostname, and then manipulate the value to remove the domain and numbers:

# cluster_name.rb
require 'socket'
Facter.add('cluster_name') do
  setcode do
    hostname = Socket.gethostname
    hostname.sub!(/..*$/, '') # remove the first period and everything after it
    hostname.gsub(/[0-9]/, '') # remove every number and return revised name
  end
end

For more optimization, you could simply use the hostname fact. You can acquire the value of an existing fact using Facter.value('factname'). Here is a simpler example starting with the existing fact:

# cluster_name.rb
Facter.add('cluster_name') do
  setcode do
    hostname = Facter.value(:hostname).sub(/..*$/, '')
    hostname.gsub(/[0-9]/, '')
  end
end

Plugins including facts are synced from the modules to the node (pluginsync) during Puppet’s initial configuration phase, prior to the catalog being built. This makes the fact available for immediate use in manifests and templates.

You can run puppet facts to see all facts, including those that have been distributed via pluginsync:

[vagrant@client facter]$ puppet facts find |grep cluster_name
    "cluster_name": "client",

# cluster_name.rb
Facter.add('cluster_name', :sleep, :timeout => 5) do
  setcode do
    Facter::Core::Execution.exec("/bin/hostname -s |/usr/bin/sed -e 's/d//g'")
  end
end

# cluster_name.rb
Facter.add('cluster_name') do
  confine 'domain' => 'example.com'
  setcode do
ruby code which provides the value...

# debian_fact.rb
Facter.add('debian_fact') do
  confine 'operatingsystem' => %w{ Debian Ubuntu }
  setcode do
ruby code which provides the value...

# configured_hostname.rb
Facter.add('configured_hostname') do
  has_weight 10
  setcode do
    if File.exist? '/etc/hostname'
      File.open('/etc/hostname') do |fh|
        return fh.gets
      end
    end
  end
end

Facter.add('configured_hostname') do
  confine "os['family']" => 'RedHat'
  has_weight 5
  setcode do
    if File.exist? '/etc/sysconfig/network'
      File.open('/etc/sysconfig/network').each do |line|
        if line.match(/^HOSTNAME=(.*)$/)
          return line.match(/^HOSTNAME=(.*)$/)[0]
        end
      end
    end
  end
end

Facter.add('fact_name', :type => :aggregate) do
  chunk('any-name-one') do
    ruby code
  end

  chunk('any-name-two') do
    different ruby code
  end

  aggregate do |chunks|
    results = Array.new
    chunks.each_value do |partial|
      results.push(partial)
    end
    return results
  end
end

Debugging

Your new fact will not appear in the output of facter. Use puppet facts find to see the values of custom Puppet facts.

To test the output of your fact, run puppet facts find --debug to see verbose details of where facts are being loaded from:

$ puppet facts find --debug
...
Debug: Loading facts from modules/stdlib/lib/facter/facter_dot_d.rb
Debug: Loading facts from modules/stdlib/lib/facter/pe_version.rb
Debug: Loading facts from modules/stdlib/lib/facter/puppet_vardir.rb
Debug: Loading facts from modules/puppet/facts.d/three_simple_facts.txt

If the output from your custom fact wasn’t in the proper format, you’ll get errors like this:

Fact file facts.d/python_sample.py was parsed but returned an empty data set

In those situations, run the program by hand and examine the output. Here’s the example I used to generate the preceding complaint:

$ /etc/puppetlabs/code/environment/testing/modules/puppet
$ facts.d/python_sample.py
fact1=this
fact2
fact3=that

Understanding Implementation Issues

There are a few things to understand about how the Puppet agent utilizes facts:

• External facts are evaluated first, and thus cannot reference or use Facter or Ruby facts.
• Ruby facts are evaluated later and can use values from external facts.
• External executable facts are forked instead of executed within the same process. This can have performance implications if thousands of external fact programs are used.

Outside of these considerations, the facts created by these programs are equal and indistinguishable.

Puppet Functions

New to Puppet 4 is the ability to write functions in the Puppet configuration language. This gives you the freedom to write functions for use in your manifests entirely in the Puppet language.

Each function should be declared in a separate file stored in the functions/ directory of a module. The file should be named for the function name followed by the .pp extension. Each function defined in a module must be qualified within the module’s namespace.

A function placed in an environment’s manifests/ directory should be qualified with the environment:: namespace.

For an example, we’ll create a function called make_boolean() that accepts many types of input and returns a Boolean value. Much like the str2boolean and num2boolean functions from the puppetlabs/stdlib library, this function will accept either strings (yes/no/y/n/on/off/true/false) or numbers. Unlike the stdlib functions, it will handle either input type, and also accept Boolean input without raising an error.

The function should be placed in a file named functions/make_boolean.pp in the module directory. The function must be named with the qualified scope of the module. For our example, the function would be declared like so:

# functions/make_boolean.pp
function puppet::make_boolean( Variant[String,Numeric,Boolean] $inputvalue ) {
  case $inputvalue {
    Undef:   { false }
    Boolean: { $inputvalue }
    Numeric: {
      case $inputvalue {
        0:       { false }
        default: { true }
      }
    }
    String: {
      case $inputvalue {
        /^(?i:off|false|no|n|'')$/: { false }
        default: { true }
      }
    }
  }
}

Functions written in the Puppet language can only take actions possible within the Puppet language. Ruby functions remain significantly more powerful.

Ruby Functions

Each Ruby function should be declared in a separate file, stored in the lib/puppet/functions/modulename/ directory of the module, and named for the function followed by the .rb extension.

Define the function by calling Puppet::Functions.create_function() with the name of the new function as a Ruby symbol for the only parameter. The code for the function should be defined within a method of the same name.

Our make_boolean() function from the previous section could look like this:

Puppet::Functions.create_function(:'puppet::make_boolean') do
  def make_boolean( value )
    if [true, false].include? value
      return value
    elsif value.nil?
      return false
    elsif value.is_a? Integer
      return value == 0 ? false : true
    elsif value.is_a? String
      case value
      when /^s*(?i:false|no|n|off)s*$/
        return false
      when ''
        return false
      when /^s*0+s*$/
        return false
      else
        return true
      end
    end
  end
end

Puppet::Functions.create_function(:'puppet::make_boolean') do
  dispatch :make_boolean_from_string do
    param 'String', :value
  end

  dispatch :make_boolean_from_integer do
    param 'Integer', :value
  end

  dispatch :return_value do
    param 'Boolean', :value
  end

  dispatch :return_false do
    param 'Undef', :false
  end

  def return_value( value )
    return value
  end

  def return_false( value )
    return false
  end

  def make_boolean_from_integer( value )
    return value == 0 ? false : true
  end

  def make_boolean_from_string( value )
    case value
    when /^s*(?i:false|no|n|off)s*$/
      return false
    when ''
      return false
    when /^s*0+s*$/
      return false
    else
      return true
    end
  end
end

Puppet::Functions.create_function(:'puppet::find_largest') do
  dispatch :compare_two_values do
    required_param 'Integer', :first
    optional_param 'Integer', :second
  end

  dispatch :compare_unlimited_values do
    repeated_param 'Integer', :values
  end

  def compare_two_values( first, second )
    ...
  end

  def compare_unlimited_values( *values )
    ...
  end

require 'ipaddr'
# @param [String] address - an IPv4 or IPv6 address
# @param [String] netmask - a netmask
Puppet::Functions.create_function(:'puppet::get_subnet') do
  def get_subnet( address, netmask )
    if !address.nil?
      ip = IPAddr.new( address )
      return ip.mask( netmask ).to_s
    end
  end
end

puppet::get_subnet( $facts['networking']['ip'], $facts['networking']['netmask'] )

Puppet::Functions.create_function(:'mymodule::outer_function') do
  def outer_function( host_name )
    call_function('process_value', ['hostname', host_name] )
  end
end

Puppet::Functions.create_function(:'mymodule::outer_function') do

  def outer_function( fact_value )
    raise Puppet::ParseError, 'Fact not available!' if fact_value.nil?
    ...things you do if the fact is available...
  end
end

Using Custom Functions

Whether your function was written in Puppet or Ruby, you can use a function you’ve created the same way as a function built into Puppet. For example, we could use the make_boolean() function we’ve defined to ensure that a service receives a Boolean value for enable, no matter what type of value was passed to it:

service { 'puppet':
  ensure    => $status,
  enable    => puppet::make_boolean( $enabled ),
  subscribe => Package['puppet-agent'],
}

Within Puppet templates, all functions are methods of the scope object. Use the call_function() method. This is because, as you might guess, templates are parsed within the scope of the Ruby function template(). Use square brackets around the input variables to create a single array of parameters:

true_or_false = <%= scope.call_function('puppet::make_boolean', [inputvar]) %>

Defining Ensurable

Most types are ensurable, meaning that we compare the current state of the type to determine what changes are necessary. Add the method ensurable to the type:

Puppet::Type.newtype( :elephant ) do
  ensurable
end

The provider for the type is required to define three methods to create the resource, verify if it exists, and destroy it.

Accepting Params and Properties

There are two types of values that can be passed into a type with an attribute. params are values used by the type but not stored or verifiable on the resource’s manifestation.

Every type must have a namevar parameter, which is how the resource is uniquely identified on the system. For user resources this would be the uid, while file resources use path. For our example, the unique identifier is the elephant’s name:

  newparam( :name, :namevar => true ) do
    desc "The elephant's name"
  end

A property is different in that we can retrieve the property from the resource and compare the values:

  newproperty( :color ) do
    desc "The color of the elephant"
    defaultto 'grey'
  end

  newproperty( :motto ) do
    desc "The elephant's motto"
  end

The preceding example defines the properties and sets a default value for the color.

Validating Input Values

You can perform input validation on each param or property provided to the type. For example, we need to ensure that the color is a known elephant color:

  newproperty( :color ) do
    desc "The color of the elephant"
    defaultto 'gray'

    validate do |value|
      unless ['grey','brown','red','white'].include? value
        raise ArgumentError, "No elephants are colored #{value}"
      end
    end
  end

There’s a newvalues() method that provides this kind of test, albeit without the more informative message. If a clear error message is not required, it’s much shorter to write:

  newproperty( :color ) do
    desc "The color of the elephant"
    defaultto 'grey'
    newvalues('grey','brown','red','white')
  end

  newproperty( :motto ) do
    desc "The elephant's motto"
    newvalues(/^[ws'.]$/)
  end

The latter form used a Regexp to accept any string that contained only alphanumeric letters, spaces, single quotes, and periods.

You can munge values to provide clean mapping for local inconsistencies:

  newproperty( :color ) do
    desc "The color of the elephant"
    defaultto 'grey'
    newvalues('grey','brown','red','white')
    munge do |value|
      case value
      when 'gray'
        'grey'
      else
         super
      end
    end
  end

You can perform input validation for the entire type using a global validate method. The input values are available as attributes of self:

Puppet::Type.newtype( :elephant ) do
  ...

  validate do |value|
    if self[:motto].nil? and self[:color].nil?
      raise ArgumentError, "Both a color and a motto are required input."
    end
  end
end

You can define a pre_run_check method, which will run the code for each instance added to the catalog just before attempting to apply the catalog on the node. Every instance that generates an error will be output as an error by Puppet, and the Puppet run will be aborted.

Keep in mind the difference in the placement of these methods in the run cycle:

• The validate method is called each time a resource of this type is added to the catalog. Other resources and types may not yet be parsed and available in the catalog yet.
• The pre_run_check method is called after the entire catalog is built, and every instance should exist within the catalog. This is the only valid place to check that requirements for this type exist within the catalog.

Defining Implicit Dependencies

As you might recall from Chapter 7, many Puppet types will autorequire other items in the catalog upon which they depend. For example, a file resource will autorequire the user who owns the file. If the user exists in the catalog, that file resource will depend on that user resource.

You can autorequire dependent resources for your type as shown here:

  Puppet::Type.newtype( :elephant ) do

  autorequire(:file) do
    '/tmp/elephants'
  end
  autorequire(:file) do
    '/tmp'
  end

As our elephant exists in the /tmp/elephants directory, we autorequire the File['/tmp'] resource. If that resource exists in the catalog, we will depend on it. If it is not defined, then the dependency will not be set.

In addition to autorequire, you can use the same syntax for autobefore, autonotify, and autosubscribe. These create soft dependencies and refresh events in the same manner as the ordering metaparameters without auto in their name.

Learning More About Puppet Types

One of the most informative ways to debug issues with Puppet types is to run Puppet with the --debug argument. The debugging shows the loading of custom types and selection of the provider.

The best book to learn more about Puppet types is Dan Bode and Nan Liu’s Puppet Types and Providers. That book covers this topic much more exhaustively than this quick introduction.

The Puppet::Type documentation provides a detailed reference to all methods available.

The Custom Types documentation provides some prescriptive guidance.

Determining Provider Suitability

Each provider needs to define ways that Puppet can determine if the provider is suitable to manage that resource on a given node.

There are a wide variety of suitability tests. Following are some examples with comments about their use:

Puppet::Type.type( :elephant ).provide( :posix ) do
  # Test the operating system family fact
  confine :osfamily => ['redhat','debian','freebsd','solaris']

  # Test true/false comparisons
  confine :true     => /^4/.match( clientversion )

  # A directory for /tmp must exist
  confine :exists => '/tmp'

  # Ensure that the 'posix' feature is available on this target
  confine :feature => 'posix'
end

Assigning a Default Provider

At times, multiple providers can manage the same resource on a given node. For example, the yum and rpm providers can both manage packages on a CentOS node.

You can declare the provider suitable to be the default provider for a given platform by using defaultfor method and passing it a fact name as a symbol, and an array of suitable values:

  # Test the operating system family fact
  defaultfor :osfamily => ['redhat','debian','freebsd','solaris']

Defining Commands for Use

The commands method lets you test for the existence of a file, and sets up an instance method you can use to call the program. If the command cannot be found, then the provider is not suitable for this resource:

  # The echo command must be within /bin
  commands :echo => '/bin/echo'

  # the ls command must be found in Puppet's path
  commands :ls  => 'ls'

The commands method also defines a new method named for the command that invokes the command, passing all arguments as space-separated command-line options. The method puts the command invocation in the Puppet debug output, and it automatically traps nonzero exit codes and raises Puppet::ExecutionFailure for you.

This is significantly better than using Ruby’s built-in command execution methods, and having to write the exception handling yourself.

Ensure the Resource State

Most types are ensurable, meaning that the provider must validate the existence of the resource and determine what changes are necessary. Define three methods to create the resource, verify its existence, and destroy it:

Puppet::Type.type( :elephant ).provide( :posix ) do
  # where elephants can be found
  filename = '/etc/elephants/' + resource['name']

  # commands we need.
  commands :echo => 'echo'
  commands :ls   => 'ls'
  commands :rm   => 'rm'

  # ensurable requires these methods
  def create
    echo("color = #{resource['color']}",'>',filename)
    echo("motto = #{resource['motto']}",'>>',filename)
  end

  def exists?
    begin
      ls(filename)
    rescue Puppet::ExecutionFailure => e
      false
    end
  end

  def destroy
    rm(filename)
  end

These three methods are used to handle transition from present to absent and vice versa. The type calls the exists? method and then determines whether to call create or destroy in order to meet the state defined by ensure.

Adjusting Properties

For each property with a value that needs to be compared to the resource, you’ll need to define two methods—a getter and a setter for each attribute:

  # where elephants can be found
  filename = '/etc/elephants/' + resource['name']

  # commands we need.
  commands :sed => 'sed'

  def color
    sed('-e','s/^color = (.*)$/1/',filename)
  end

  def color=(value)
    sed('-i','-e','s/^color = /color = #{value}/',filename)
  end

The first method retrieves the current value for the property color, and the second method changes the elephant’s color on the node. The definition for motto would be nearly identical.

If there are many attributes that change values, you may want to cache up the changes and write them out at once. After calling any setter methods, a resource will call the flush method if defined:

  def color=(value)
    true
  end

  def flush
    echo("color = #{resource['color']}",'>',filename)
    echo("motto = #{resource['motto']}",'>>',filename)
    @property_hash = resource.to_hash
  end

If the resource was modified through a long command line of values (e.g., usermod), you could track which values were changed and build a customized command invocation with only those values. Because this resource is only two lines of text, it’s significantly easier to just write the file out again.

The final line caches the current values of the resource into an instance variable. Let’s talk about what can be done with caching now.

Providing a List of Instances

If it is low impact to read in the resources, you can implement an instances class method that will load all instances into memory. This can improve performance in comparison to loading each one separately and modifying it.

Resource providers can make use of that data—for example, when someone uses the following command:

$ puppet resource elephant

To disable preloading of instance data, define it with an empty array.

  self.instances
    []
  end

To preload instance data, we simply need to construct the method to output each file in the directory and create a new object with the values:

  commands :ls  => 'ls'
  commands :cat => 'cat'

  self.instances
    elephants = ls('/tmp/elephants/)
    # For each elephant...
    elephants.split("
").collect do |elephant|
      attrs = Hash.new
      output = cat("/tmp/elephants/#{elephant}")
      # for each line in the file
      output.split("
").collect do |line|
        name, value = line.split(' = ', 2)
        # store the attribute
        attrs[name] = value
      end
      # add the name and its status and then create the resource type
      attrs[:name] = elephant
      attrs[:ensure] = :present
      new( attrs )
    end
  end

The preceding code reads each assignment in the elephant file, and assigns the value to the name in a hash. It adds the elephant’s name to the hash, and sets ensure to present. Voilà, we have built this resource in memory! This preloads every instance of the resource, and makes the data available in the @property_hash instance variable.

When the instances method is available, puppet agent and puppet apply will load all instances in this manner, and then match up resources in the database to the provider that returned their values.

Taking Advantage of Caching

If all of your instances are cached in memory, then you don’t need to read from disk every time. This means you can rewrite your exists method to simply...

  def exists?
    @property_hash[:ensure] == 'present'
  end

Although in the same sense, you need to ensure that any resources created or deleted are updated in memory:

  def create
    echo("color = #{resource['color']}",'>',filename)
    echo("motto = #{resource['motto']}",'>>',filename)
    @property_hash[:ensure] = :present
  end

  def destroy
    rm(filename)
    @property_hash[:ensure] = 'absent'
  end

Finally, how about each of those instance setter and getter methods? These would each be identical, as they would be just setting or reading from the hash. There is a convenience method, mk_resource_methods, which would define all resource attribute methods as follows:

  def color
    @property_hash[:color] || :absent
  end
  def color=(value)
    @property_hash[:color] = value
  end

Place the mk_resource_methods invocation near the top of the provider. You can then override any one or more of the default methods it creates. On a resource type with 20 or more attributes, this convenience method will save you a lot of typing!

However, now that you are only saving the changes back to a hash, you must define a flush() method to write the changes out to disk as described in “Adjusting Properties”.

Learning More About Puppet Providers

One of the most informative ways to debug issues with Puppet types is to run Puppet with the --debug argument. The debugging shows the selection and execution of the provider for a given node.

The best book to learn more about Puppet providers is Dan Bode and Nan Liu’s Puppet Types and Providers. That book covers providers much more exhaustively than this quick introduction.

The Puppet::Provider documentation provides a detailed reference to all methods available.

The Provider Development documentation includes many details of creating providers.

Using Data from Hiera

Create a module data configuration file that defines the Hiera hierarchy. The file should be named hiera.yaml, much like the global Hiera configuration file, but it uses a v5 configuration format.

The file must contain a version key with value 5, The hierarchy in the file must be an array of hashes that define the data sources. The datadir used by each source will be a path relative to and contained within the module root. An example file is shown here:

---
version: 5
defaults:
  datadir: data
  data_hash: yaml_data

hierarchy:
  - name: "OS family"
    backend: json_data
    path: "os/%{facts.os.family}.json"

  - name: "common"
    path: "common.yaml"

Create the data/ directory indicated in the file, and populate Hiera data files with data specific to this module.

Puppet::Functions.create_function(:'specialapp::data') do
  def data() 
    # the following is just example code to be replaced
    # Return a hash with parameter name to value mapping for the user class
    return {
      'specialapp::user::id'   => 'Value for parameter $id',
      'specialapp::user::name' => 'Value for parameter $name',
    }
  end
end

  - name: "module data"
    lookup_key: specialapp::data

Performing Lookup Queries

For the example specialapp module, the new lookup strategy of global, then environment, then module data providers, would be queried as follows if you used the function data source for the module:

class specialapp(
  Integer $id,  # will check global Hiera, then environment data provider,
  String $user, # then call specialapp::data() to get all values
) {

If you used the hiera data source for the module, then parameter values would be independently looked up in each data source:

class specialapp(
  Integer $id,  # will check global Hiera, then environment data provider,
  String $user, # then check module Hiera data
) {