Published on 12 July 2012 by

We’ve written several blog posts about testing modules. There’s good reason for this: testing is an increasingly important part of module development here at Puppet Labs. Comprehensive module test suites help us avoid regressions and support a wide range of Puppet and operating system versions. Like everything we do, we want the work that we’ve put into testing to be of as much benefit to the Puppet community as possible.

Despite all this prior work, module testing has remained somewhat painful. There’s a lot of boilerplate to write and, for complex modules, there can be a number of dependencies which need to be setup correctly. These dependencies make life especially difficult when trying to configure automated testing—every module would need its own testing script.

To resolve this, we’ve been working on a common set of testing helpers which any module author can use to abstract away that common boilerplate. These helpers take the form of a Ruby gem called puppetlabs_spec_helper. The source code for this gem is available on Github.

This gem provides a few things:

  • A common helper to reduce the boilerplate needed to use rspec-puppet, a unit-testing framework for puppet modules.
  • A set of Rake tasks that handle common testing and development tasks including:
    • Installing and cleaning up test dependencies
    • Building a module package for the Puppet Forge
    • Verifying best practices with puppet-lint

The rest of this blog post assumes you’re at least somewhat familiar with module testing and rspec-puppet. If you’re not, I recommend you read the blog posts linked above and then come back here.

A Simple Module

To demonstrate the puppetlabs_spec_helper gem, we’re going to add tests to a simple module that installs apache. Let’s start by creating that skeleton module. Create a file in modules/apache/manifests/init.pp with the following contents:

class apache (
  $package_name = false
){
  if is_string($package_name) {
    $real_package_name = $package_name
  } else {
    case $::osfamily {
      'RedHat': {
        $real_package_name = 'httpd'
      }
      'Debian': {
        $real_package_name = 'apache2'
      }
      default: {
        fail("unsupported osfamily: $::osfamily")
      }
    }
  }

  package { 'httpd':
    ensure => installed,
    name   => $real_package_name
  }

  service { 'httpd':
    ensure => running,
    name   => $real_package_name
  }
}

This manifest is designed to demonstrate both important situations for writing module spec tests:

  • Are there any conditionals? If so, tests should be written to verify that all branches are correctly taken.
  • Are there any forwarded parameters? It’s important to ensure that your code accurately passes parameters to contained classes and types.

Testing the Module

Now that we have a module, it’s time to test it. First, let’s make sure puppetlabs_spec_helper is available to us. At the time of this writing, the latest version is 0.2.0.

% gem install puppetlabs_spec_helper

We’ll be adding several more files to the project. If your module was created using the Puppet Module Tool, it may already contain some of these files; you will need to replace them with the versions described here in order for the puppetlabs_spec_helper to work correctly. If your module does not contain these files you will need to create them manually.

modules/apache/Rakefile
modules/apache/.fixtures.yml
modules/apache/spec/spec_helper.rb
modules/apache/spec/classes/apache_spec.rb

This is where using puppetlabs_spec_helper diverges from the traditional rspec-puppet setup. If you’ve tested a module with rspec-puppet before you will notice a new file—.fixtures.yml. This file is used by the common rake tasks to automatically install dependencies for test runs. For our example, its contents should be:

fixtures:
  repositories:
    stdlib: git://github.com/puppetlabs/puppetlabs-stdlib.git
  symlinks:
    apache: "#{source_dir}"

This defines a testing environment with two modules: one called ‘stdlib’ fetched from a Github repository, and one called ‘apache’ (our module) which will be a symlink to the current source directory.

Another difference between traditional rspec-puppet code and the puppetlabs_spec_helper gem is the boilerplate. Both the Rakefile and spec/spec_helper.rb are provided by the gem, and should be two-liners in your module:

Rakefile:

require 'rubygems'
require 'puppetlabs_spec_helper/rake_tasks'

spec/spec_helper.rb:

require 'rubygems'
require 'puppetlabs_spec_helper/module_spec_helper'

It is possible to add additional functionality to these files if you desire. Whenever adding new functionality you should try to decide if it truly is module-specific. If it could be useful to other modules the best option is to open a pull request against puppetlabs_spec_helper.

Using the Rakefile

Before we continue, let’s see which Rake tasks are available to us:

% rake help
rake build            # Build puppet module package
rake clean            # Clean a built module package
rake coverage         # Generate code coverage information
rake help             # Display the list of available rake tasks
rake lint             # Check puppet manifests with puppet-lint
rake spec             # Run spec tests in a clean fixtures directory
rake spec_clean       # Clean up the fixtures directory
rake spec_prep        # Create the fixtures directory
rake spec_standalone  # Run spec tests on an existing fixtures directory
  • 'build' and 'clean' are for creating module packages for the Puppet Forge. They require a Modulefile, which is missing from this example.
  • 'coverage' runs rcov on your spec tests; it is only useful if your module includes Ruby extensions to Puppet.
  • 'help' displays the available rake tasks.
  • 'lint' runs puppet-lint against your module.
  • 'spec' will download your testing dependencies, run your rspec code, and finally clean up the test directory.
  • The remaining three tasks allow you to handle setup, teardown, and testing as individual steps.

Writing the Test Code

Finally it’s time to write the test cases for our module. As discussed earlier, our example module is designed to demonstrate the important testing points. We will need to write several tests to cover them:

  • If a package name is provided, it should be used
  • If a package name is not provided AND we are on a supported osfamily, a default name should be used.
    • Since there are two supported osfamilies, we should write tests for both of them.
  • If a package name is not provided AND we are not on a supported osfamily, the catalog compilation should fail.

This is a total of four fairly simple test cases. Writing these tests is exactly the same as if you were using rspec-puppet without the puppetlabs_spec_helper gem. Their implementation will go in apache_spec.rb:

require 'spec_helper'

describe 'apache', :type => 'class' do

  context "On a Debian OS with no package name specified" do
    let :facts do
      {
        :osfamily => 'Debian'
      }
    end

    it {
      should contain_package('httpd').with( { 'name' => 'apache2' } )
      should contain_service('httpd').with( { 'name' => 'apache2' } )
    }
  end

  context "On a RedHat OS with no package name specified" do
    let :facts do
      {
        :osfamily => 'RedHat'
      }
    end

    it {
      should contain_package('httpd').with( { 'name' => 'httpd' } )
      should contain_service('httpd').with( { 'name' => 'httpd' } )
    }
  end

  context "On an unknown OS with no package name specified" do
    let :facts do
      {
        :osfamily => 'Darwin'
      }
    end

    it {
      expect { should raise_error(Puppet::Error) }
    }
  end

  context "With a package name specified" do
    let :params do
      {
        :package_name => 'apache'
      }
    end

    it {
      should contain_package('httpd').with( { 'name' => 'apache' } )
      should contain_service('httpd').with( { 'name' => 'apache' } )
    }
  end
end

We’re now ready to run our tests:

% rake spec
Cloning into spec/fixtures/modules/stdlib...
remote: Counting objects: 1988, done.
remote: Compressing objects: 100% (742/742), done.
remote: Total 1988 (delta 1012), reused 1905 (delta 943)
Receiving objects: 100% (1988/1988), 267.21 KiB | 134 KiB/s, done.
Resolving deltas: 100% (1012/1012), done.
/Users/branan/.rvm/rubies/ruby-1.8.7-p358/bin/ruby -S rspec spec/classes/apache_spec.rb --color
....

Finished in 0.39344 seconds
4 examples, 0 failures

As you can see, the spec task automatically cloned the puppetlabs-stdlib repository before running the tests.

Continuous Integration

This automatic handling of module dependencies makes it very easy to test modules in continuous integration systems. We do this ourselves using Jenkins for a number of modules.

Using Bundler

While not strictly required, bundler makes it easy to test your module against several versions of Ruby and Puppet. Most CI systems expose test axes as environment variables, and a specially written Gemfile will allow you to use those environment variables to install the correct versions of dependencies. The Gemfile used by Puppet Labs for our modules is:

source :rubygems

puppetversion = ENV.key?('PUPPET_VERSION') ? "= #{ENV['PUPPET_VERSION']}" : ['>= 2.7']
gem 'puppet', puppetversion
gem 'puppetlabs_spec_helper', '>= 0.1.0'
Aside: Gemfiles and the Puppet Module ToolIn our modules, we name our Gemfiles .gemfile instead of Gemfile. Dotfiles are automatically ignored when packaging with the Module Tool. We recommend this practice in order to avoid clutter on end-users’ systems, but it is not strictly required.

Testing on Jenkins

If you are using the Gemfile above, your jenkins configuration should contain an axis called PUPPET_VERSION with the values being the Puppet versions you wish to use for testing.

In our environment, we use RVM to isolate test runs in their own gemsets. This is optional, and if you have a different method of isolating test runs you can safely remove the references to rvm from the test script below. $ruby is a Jenkins axis containing the various Ruby versions we test against.

#!/bin/bash

[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"

rvm use $ruby@$$ --create
gem install bundler
bundle install --gemfile=.gemfile
rake spec
RESULT=$?
rvm --force gemset delete $$
exit $RESULT

This script uses the Gemfile described in the previous section to populate a temporary unique gemset with the testing dependencies. It then runs the tests and cleans up the gemset.

Testing on Travis CI

If you do not have access to a Jenkins installation, Travis CI provides an environment that is compatible with testing Puppet modules. Unlike Jenkins, Travis will automatically handle setting up a Ruby environment and running bundler on the .gemfile. Thus a special script is not needed; we can instead use a simple configuration file. Travis requires this file be named .travis.yml.

language: ruby
rvm:
  - 1.8.7
script: "rake spec"
branches:
  only:
    - master
env:
  - PUPPET_VERSION=2.7.17
  - PUPPET_VERSION=2.7.13
  - PUPPET_VERSION=2.7.9
  - PUPPET_VERSION=2.6.9
notifications:
  email: false
gemfile: .gemfile

A detailed explanation of Travis configuration files is outside the scope of this article. If you’re unfamiliar with the Travis Yaml configuration files you should read the documentation.

Conclusion

Puppetlabs_spec_helper is part of our larger goal of encouraging testing on Puppet modules. The standardized rake tasks and common boilerplate enable module authors and contributors to easily write and run tests for Puppet code and the automation features make it easy to run tests in CI environments. Our hope is that as the tooling and process around module testing improve we will see tests become a more ubiquitous part of module development both here at Puppet and in the community at large.

Learn More:

Share via:
Posted in:

The formatting of the rakefile still appears to have consumed the contents of the spec file.

```
require 'rubygems'
require 'puppetlabs_spec_helper/rake_tasks'

spec/spec_helper.rb:

require 'rubygems'
require 'puppetlabs_spec_helper/module_spec_helper'
```

Should be

```
require 'rubygems'
require 'puppetlabs_spec_helper/rake_tasks'
```
spec/spec_helper.rb:
```
require 'rubygems'
require 'puppetlabs_spec_helper/module_spec_helper'
```

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.