Junking the Ruby DSL

Puppet 2.6 introduced the Ruby DSL to allow Puppet resources to be declare in pure Ruby. This was intended to allow Ruby developers to utilize Ruby control features, such as iteration, that weren’t available in the Puppet configuration language at the time.

The Ruby DSL was very limited in what could be expressed. There was a development attempt to make it more powerful, but the result was unstable and likely to hang the Puppet process. As the limited Ruby subset didn’t provide significant functionality, Puppet Labs instead added iteration to the Puppet language.

The Ruby DSL was deprecated in Puppet 3, and has been completely removed in Puppet 4.

Upgrading Config Environments

Unless you are dealing with an ancient Puppet environment, I doubt you have configuration file environments. These were environments defined by sections in the Puppet configuration file (usually /etc/puppet/puppet.conf):

[test]
   settings for test environment

[dev]
   settings for dev environment

These have been deprecated for many years, and won’t work at all in Puppet 4. Replace these with the much more flexible directory environments by following this process:

1. mkdir -p /etc/puppetlabs/code/environments/production.
2. Add environmentpath = /etc/puppetlabs/code/environments to the [main] section of puppet.conf.
3. Create a directory for each environment in the preceding path.
4. Move module path settings from puppet.conf to an environment.conf file in each environment’s directory.
5. Copy manifests used by each environment to the manifests/ directory of the environment.
6. Move modules used by specific environments to the modules/ directory of the environment.
7. Move modules used by all environments to the $confdir/modules/ directory, or a directory specified with the basemodulepath configuration option.

This may not cover every situation, but it will get you most of the way.

Chapter 29 covers the setup and usage of directory environments in greater detail.

Removing Node Inheritence

The only way to apply a list of classes to similar nodes in Puppet 2 was via explicit node assignments in the manifests/site.pp/ file. To avoid repeating common definitions required creating node assignments that inherited from other node assignments:

node default {
  class { 'ntp': }
  class { 'puppet::agent': }
}
node 'webserver' inherits default {
  class { 'apache': }
}

The ability to assign classes using external node classifiers (ENCs) and Hiera data provides significantly more flexible solutions. Node assignment remains possible in Puppet 4, but node inheritance has been completely removed.

You’ll learn to flexibly group class assignments to nodes in “Assigning Modules to Nodes”.

Disabling puppet kick

The limited and broken puppet kick is gone, replaced by the significantly more powerful MCollective agent for Puppet.

If you have been using puppet kick, you should disable the network service that listens on the port, as follows:

1. Remove listen = true from the puppet.conf configuration file.
2. Remove any firewall configuration that allowed connections to tcp port 8139.
3. Remove permissions for /run from the auth.conf file.

You’ll learn the powerful new way to kick off Puppet runs in Chapter 30.

Qualifying Relative Class Names

Previous versions of Puppet would sometimes resolve class names using the current context. For example, the following statement could be “intercepted”:

class mymodule {
  include ntp
}

If the class or defined type mymodule::ntp existed, it would be used instead of the global class ::ntp. This code dates far back to when Puppet modules were first being developed, and has always confused people.

Losing the Search Function

Relative namespace problems were made even more confusing by the use of the search() function to add lookup paths.

Replacing Import

The import statement has been removed:

# this won't work with Puppet 4
import commonfunctions.pp

This was previously used to pull in another manifest within the current code context. This was never a good idea.

Place the code you were importing in a module class, and include the class.

You’ll learn to create Puppet modules in Chapter 13.

Documenting Modules with Puppet Strings

The puppet doc command now only documents Puppet commands. Module documentation once provided by puppet doc has been replaced by puppetlabs-strings, which generates a complete documentation tree for the module in Markdown format.

You’ll learn the powerful new features available for documentation in Chapter 16.

Installing the Tagmail Report Processor

The tagmail Puppet report processor is no longer included by default. Instructions for installing this report processor with the optional Puppet module can be found in “Processing Puppet Node Reports”.

Querying PuppetDB

Some features have been removed from Puppet and migrated to the PuppetDB service. These include:

• The ActiveRecord storeconfigs
• The inventory service
• The puppet facts upload command (which utilized the inventory service)
• The puppet facts find --terminus rest command (which utilized the inventory service)

Adjust manifests that need these features to use the PuppetDB API: Facts Endpoint.

PuppetDB is a full-featured product worthy of its own book. At this time, the best available documentation can be found at “PuppetDB Overview” on the Puppet docs site.

Validating Variable Names

Variable names are now limited to lowercase letters, numbers, and underscores. Variable names must contain at least one letter, and start with a lowercase letter or underscore:

$5 = 'hello'              # invalid
$5letters = 'hello'       # invalid
$five_letters = 'hello'   # valid

Error: Illegal numeric variable name,
  The given name '5letters' must be a decimal value if it starts with a digit 0-9

Parameters in classes and defined resource types have the same limitations:

define mytype( $21, $32 ) {             # invalid
define mytype( $twenty1, $thirty2 ) {   # valid

Quoting Strings

Bare word strings must start with a letter. Older Puppet code contains fairly common usage of unquoted values (present, true, etc.). Previous versions of Puppet would treat these as unquoted strings, but Puppet 4 supports many data types. It is best to quote all string values to ensure they are handled properly:

$myvar = fourguys     # assumed to be String
$myvar = 42           # assumed to be Numeric (Integer)
$myvar = true         # assumed to be Boolean
$myvar = '4guys'      # quotes ensure String data type

Preventing Numeric Assignment

In previous versions of Puppet, numbers were really just strings. Unquoted numbers were unquoted strings, which happened to work. Documented best practice in Puppet 3 was to quote all numbers to ensure they were explicitly strings, but a lot of modules have unquoted numbers in them.

In Puppet 4, unquoted numerals are the Numeric data type. Numbers are validated as part of the catalog build, and an invalid number will cause the catalog build to fail:

Error: Could not parse for environment production: Illegal number '4guys'

To avoid catalog build failures, quote strings that may be misinterpreted as numbers. Some data uses hex or octal values with suffixes specific to the application. The following are situations where unquoted numbers would work in Puppet 3 but cause errors in Puppet 4:

# leading 0x causes evaluation as hexidecimal
$address =  0x1AH      # 'H' is not a valid hex number
$address = '0x1AH'     # safe as quoted string

# leading zeros cause numbers to evaluated as octal
$leadingzero =  0119   # ERROR octal has 8 bits, 0-7
$leadingzero = '0119'  # safe as quoted string

# mixed letters and numbers
$mixed_chars = 1x2x3   # will be mistaken for decimal and raise error
$mixed_chars = '1x2x3' # safe as quoted string.

Any unquoted string that starts with a number will be validated as if it were a number, and may cause a catalog build failure.

The easiest way to avoid confusion is to always quote strings. As you can pass a string containing a number as input to anything that will accept a number, it is safest to quote numbers that may be misinterpreted.

file { '/tmp/testfile.txt':
   mode => '0644',
}

Testing Boolean Facts

In previous versions of Puppet, boolean values were internally converted to the strings true and false. You could compare either the bare word or the quoted string value successfully.

Puppet 4 introduced the Boolean data type, which does not compare equally with a String value. This comparison will fail without warning:

if $is_virtual == "true" {

Search through the Puppet code for instances of quoted boolean values. Check the data source to confirm if the value is Boolean or String. For example, the is_virtual fact is now Boolean, so the check should be rewritten like so:

if $facts['is_virtual'] == true {

Empty strings evaluate to boolean true. This can trip you up if you had code that depended on a false evaluation:

# replace this
if ! $empty_string {

# with this
if $empty_string == '' {

Qualifying Defined Types

In previous versions of Puppet, it was possible and somewhat common to place defined types in the top namespace:

# modules/users/manifests/createuser.pp
define createuser(
...

This defined type could be declared exactly like a global resource type:

# any manifest
createuser{ 'jo':
  ensure => present,
  uid    => 1001,
...

This won’t work with Puppet 4, and will instead yield a nonintuitive error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
 Error while evaluating a Resource Statement, Invalid resource type createuser

This message simply means that defined types created in modules need to be prefixed with the module name:

# modules/users/manifests/createuser.pp
define users::createuser( $name, $uid, $gid ) {
  ...
}

# any manifest
users::create{ 'jo':
  ensure => present,
  uid    => 1001,
  ...

It remains possible to create top-level defined types in the manifests/ directory of an environment, but this support may be removed in a future version.

Adding Declarative Permissions

In previous versions of Puppet, if the following attributes were not defined, they were copied from the source file on the Puppet master’s filesystem:

• owner
• group
• mode

This is not very declarative, as the final state of the file depends on attributes in the filesystem you can’t read from the manifest.

If this is not possible or practical for one reason or another, add the following attribute to the resource to implement the nondeclarative behavior:

file { filename:
  source_permissions => use,
}

Removing Cron Purge

In previous versions of Puppet, the following would remove unmanaged cron entries from only the user that Puppet was running as (root):

resource { 'cron':
  purge => true,
}

This differed from most other resource purges. In Puppet 4, this will now purge unknown cron entries for every user.

Replacing MSI Package Provider

Puppet 3 deprecated the msi Windows package provider with an improved windows provider.

The following query can find all the right files for you:

$ grep -rlE 'providers*=>s*[^w]?msi[^w]?' /etc/puppet /etc/puppetlabs

Adjusting Networking Facts

With the latest version of Facter, many of the network-related facts have been restructured into hashes that allow much more intelligent retrieval and analysis of interfaces.

You may have some logic like this in existing manifests:

  if( $ipaddress_eth0 =~ /^10./ ) {
    $has_rfc1918address = true
  }

You’ll need to replace that with something much clearer. For example:

  if( $facts['networking']['interfaces']['ip'] =~ /^10./ )
    $has_rfc1918address = true
  }

That only checks the primary IP address. The complete list of IPs can be retrieved by iterating over the $facts hash, which allows you to use a lambda to test each interface:

  $net10ints = $facts['networking']['interfaces'].filter |$interface,$config| {
    $config['ip'] =~ /^10./
  }
  if( length($net10ints) > 0 ) {
    $has_rfc1918address = true
  }

Using Directory Environments

If directory environments are enabled, you can test module upgrades in place. Create an environment named upgrade environment with a complete copy of production. This may be as simple as the following:

$ cd /etc/puppet/environments 
$ cp -r production upgrade

Add the following line to the ${environmentpath}/upgrade/environment.conf file.

  parser = future

Then you can test out individual clients by changing the environment on the command line, as follows:

$ puppet apply upgrade/manifests/site.pp --environment=upgrade --noop

or on a client of a Puppet master:

$ puppet agent --test --environment=upgrade --noop

Duplicating a Master or Node

If directory environments are not available, then an alternative is to clone a Puppet master or a node that uses puppet apply. Then make the following change to the Puppet configuration file (usually /etc/puppet/puppet.conf):

[main]
  parser = future

You would test this with the same commands shown in the preceding section.

Adding else to unless

You can now use else with unless:

unless ( $somevalue > 10 ) {
   # do this
}
else {
   # do that
}

It is generally better to use if/else than unless/else for readability, but readable code is the most important thing. Use whichever reads more naturally.

Calling Functions in Strings

You can now call functions from within a double-quoted string by wrapping them in curly braces exactly as if they were variables. For all intents and purposes, the function call is a short-lived, unnamed variable:

notify { 'need_coffee':
  message => "I need a cup of coffee. Remind me in ${fqdn_rand(10)} minutes.",
}

This may allow you to remove some single-use variable assignments.

Matching String Regexps

You can now match against regular expressions defined by strings:

$what_did_you_drink !~ "^coffee$"	# uses a string value for regexp

The quoted string on the right is converted to a Regexp before matching against the left String variable. This can provide flexibility by allowing interpolated variables in a regular expression:

$you_drank =~ "^${drink_du_jour}$"	# uses a variable for matching in Regexp

Letting Expressions Stand Alone

Previous versions of Puppet required the results of all expressions to be assigned. In Puppet 4, an expression can stand alone. If it is the last expression in a lambda or a function, the result will be returned. Otherwise, the value will be discarded.

The following rather simple lambda returns the result of the last expression within the lambda block:

$tenfold = with( $input_number ) |$number| { 
  $number * 10
}

Blocks of code (lambdas and functions) always return the result of their last expression. You do not need an explicit return.

If the returned value is not used, the result is silently discarded:

# the resulting value is ignored
with( $input_number ) |$number| { 
  $number * 10
}

Chaining Assignments

You can now change multiple assignments in the same expression. You can chain both equality and addition operations, like so:

$security_deposit = 100
$first = 250
$last = 250
$down_payment = $first + $security_deposit

# could be rewritten as
$first = $last = 250
$down_payment = $first + ( $security_deposit = 100 )

As chained assignments have low precedence, you must use parentheses to ensure proper ordering.

Chaining Expressions with a Semicolon

You can now use a semicolon to concatenate two expressions together:

$fname = 'Jo'; $lname = 'Rhett'

The semicolon can be used to prevent the last statement of a block from being returned as the value:

{
  $fname = 'Jo'; $lname = 'Rhett'; 1     # returns 1 for success
}

Using Hash and Array Literals

Older versions of Puppet required you to assign arrays and hashes to variables before using them in resources or functions. Puppet 4 allows you to use literal arrays and hashes more naturally in a Puppet manifest:

notify { ['one','two','three'][1]: }    # produces the output "two"

New in Puppet 4, you can concatenate arrays and merge hashes with +:

$my_list = [1,4,7]
$bigger_list = $my_list + [14,17]                # equals [1,4,7,14,17]

$key_pairs       = {name => 'Jo', uid => 1001}
$user_definition = $key_pairs + { gid => 500 }   # hash now has name, uid, gid...

Also new in Puppet 4, you can append to arrays with <<. Watch out, as anything appended to an array is added as a single entry. This means that an array appended to an array adds an array, not multiple values, in the last position:

$my_list << 33            # equals [1,4,7,33]
$my_list << [33,35]       # equals [1,4,7,[33,35]]

Configuring Error Reporting

Error reports are much improved with Puppet 4. You’ll generally see the following improvements:

Most errors now show character position on line

Errors from manifest parsing use the format filename:line_number:character_number when reporting where an error was found:

Error: Illegal numeric variable name,
  The given name '5words' must be a decimal value
  if it starts with a digit 0-9 at /vagrant/manifests/broken.pp:2:1

Many block and token parsing errors have been improved

In previous versions, the message would often indicate something completely unhelpful to diagnose the error. A concerted effort was made in Puppet 4 to clean those up and improve clarity.

The maximum warnings or errors Puppet will display is configurable

The following settings in puppet.conf can be used to override the defaults of showing 10 of each type:

[main]
  max_errors = 3
  max_warnings = 3
  max_deprecations = 20

No matter what the limits are configured to show, a final error line provides the total error and warning count:

Error: Found 4 errors. Giving up