published on 15 August 2011

We are evaluating two tools for testing Puppet modules. One is rspec-puppet, the other is cucumber-puppet. The hope is to get standards for testing into the module forge and hook it into a CI framework such as Hudson or Jenkins. This will enable Puppet Forge users to immediately evaluate a module in terms of its test failure rate, and enable developers to see the intended behavior of the module. The higher level goal is to open a discussion among developers on the Puppet Forge about the intended behavior of modules, what they need to do, and what they should and should not be doing.

Another huge goal of this project is to develop a way to test Puppet without running puppet apply and looking to see what worked or didn’t. Both of the tools under consideration allow the developer/sysadmin to test the modules against Puppet to see if they compile (equivalent to a puppet apply —noop) and then hold the catalog object so that we can see if the catalog is formed as intended.

Cucumber-puppet is a puppetification of the gherkin-based Cucumber Behavior Driven Development (BDD) framework.

Cucumber-puppet's advantages are that it is a more mature codebase, it is faster to start with, and it uses the human-readable gherkin language for its tests. One disadvantage for cucumber-puppet is that it is difficult to contain all of the tests within a cucumber feature. Many of the tests we have written in cucumber-puppet involved creating dummy nodes in a site.pp to write cucumber tests against. Cucumber-puppet also doesn’t seem to be compatible with Puppet 2.7.x, at least not yet. It is however compatible with the 0.25.x branch which is a benefit for legacy puppet users and administrators.

Rspec-puppet is a puppetification of the rspec BDD framework.

Rspec-puppet's advantages are that it is brand new and has seen a flurry of development since its inception. Puppet already uses a lot of rspec to do testing, and, since the gherkin layer is removed, it can be coded entirely in Ruby. Also, in contrast to cucumber-puppet, rspec-puppet tests can be entirely contained within the spec test file. And for compatibility, rspec-puppet works well with the Puppet 2.6.x/2.7.x branches, but does not work with the 0.25.x branch. One note for testing both 2.6.x and 2.7.x is that 2.6.x will raise Puppet::Error exceptions for catalog compilation failures while the 2.7.x branch raises general Runtime exceptions.

Quick Getting Started With Our Examples

We have taken the puppetlabs-apt module and written fairly comprehensive tests for it in both cucumber-puppet and rspec-puppet. In this example we will set up the cucumber-puppet and rspec-puppet tools and run both suites on the puppet-apt module. Note that for this example we will be using the nibalizer fork of puppet-apt and the mlitteken forks of cucumber-puppet and rspec-puppet. For this example I started with a stock Ubuntu VM using the wonderful tool Vagrant.

Setup

Start with the following packages (and their dependencies):

Installed via apt:

  • git-core
  • rubygems1.8
  • ruby
  • ruby1.8-dev
  • libopenssl-ruby1.8

Installed via rubygems:

  • gherkin
  • cucumber
  • gem-man
  • templater
  • rspec
  • puppet (version 2.6.9

Get the testing tools:

$ mkdir src
$ cd src
$ git clone -b example https://github.com/mlitteken/rspec-puppet.git
$ git clone -b example https://github.com/mlitteken/cucumber-puppet.git
$ ls
cucumber-puppet  rspec-puppet
$ echo $RUBYLIB

$ echo "export RUBYLIB=~/src/cucumber-puppet/lib:~/src/rspec-puppet/lib:${RUBYLIB}" >> ~/.bashrc
$ echo "export RUBYPATH=~/src/cucumber-puppet/bin:${RUBYPATH}" >> ~/.bashrc
$ echo "export PATH=${PATH}:~/src/cucumber-puppet/bin" >> ~/.bashrc
$ source ~/.bashrc

Create the file structure:

$ cd
$ mkdir puppetforge/{,manifests,modules}

Clone the apt module:

$ cd puppetforge/modules
$ git clone -b example https://github.com/nibalizer/puppet-apt apt
$ cd apt

Run cucumber-puppet

$ cd ~/puppetforge/modules/apt
$ cucumber-puppet features/modules/apt/apt_*.feature

If all went well you should have seen a ton of green fly by and a summary at the end:

27 scenarios (27 passed)
258 steps (258 passed)
0m4.228s

Cucumber and cucumber-puppet use green to indicate success, red to indicate test failure, and other colors to report information about the test run.

Run rspec-puppet

$ cd ~/puppetforge/modules/apt
$ rspec --color --format documentation spec/{classes,defines}/apt*_spec.rb

If all went well you should have seen a ton of green fly by and a summary at the end:

Finished in 8.88 seconds
119 examples, 0 failures

Rspec and rspec-puppet use green to indicate success, red to indicate test failure, and other colors to report information about the test run.

Cucumber-puppet Mini Tutorial on Use

What are we testing?

The point of this very quick example is to test the apt class to see if it will create a file resource “sources.list”.

Create a very basic feature

Cucumber-puppet runs on files called testname.features; we have prepended the module name to these tests to unify our naming scheme between cucumber-puppet and rspec-puppet. We have tried to name the test files after the file in the manifests directory they are testing. Feature files go in features/modules/apt/.

Let's create a file. It will be named apt_example_init.feature because we have already defined an apt_init.feature. (The absolute path, for clarity, is: ~/puppetforge/apt/features/modules/apt/apt_example_init.feature)

Feature: example init.pp test
  In order to have an effective apt module
  The apt class needs to have certain files
  And must run some execs

    Scenario: Apt Class, basics, sanity check
    Given a node of class "apt"
    When I compile the catalog
    Then compilation should succeed
    And all resource dependencies should resolve

Run this command with:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

If all went well it should have read the scenario back to you in green, indicating success. The lines under “Feature” are fluff and are not parsed by cucumber-puppet. The lines under "Scenario" are parsed and have precise meanings to cucumber-puppet. This is a very basic scenario and it serves mostly to demonstrate that the catalog compiles and resources dependencies resolve.

Add a test to the feature

You may find it advantageous to use vim tools for cucumber.

Add the following to your apt_example_init.feature, below the first scenario. These scenarios will compile the catalog independently. Ideally they will test different things. Make sure to indent properly!

    Scenario: Basic class, sanity check
    Given a node of class "apt"
    When I compile the catalog
    Then compilation should succeed
    And all resource dependencies should resolve
    And the exec named "apt_update" should be run

And run the test:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

It errored out in yellow.

And the exec named "apt_update" should be run # features/modules/apt/apt_example_init.feature:17
      Undefined step: "the exec named "apt_update" should be run" (Cucumber::Undefined)
      features/modules/apt/apt_example_init.feature:17:in `And the exec named "apt_update" should be run'

This means we need to write a step.

Add a step definition

The previous test summarized its results as follows:

2 scenarios (1 undefined, 1 passed)
9 steps (1 undefined, 8 passed)
0m0.781s

Notice that of 2 scenarios, only 1 failed; the other passed quite elegantly. You can implement step definitions for undefined steps with these snippets:

Then /^the exec named "([^"]*)" should be run$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

This gives you a prototype step definition to match against. Cucumber-puppet works by matching the steps defined in features to regular expressions defined in ruby files in features/steps. Since we want to test an exec type we’ll add this to features/steps/exec.rb

$ vim ~/puppetforge/features/steps/exec.rb

Add the following to the file (yes, this is mostly a copy and paste operation):

Then /^the exec named "([^"]*)" should be run$/ do |arg1|
  steps %Q{
    Then there should be a resource "Exec[#{arg1}]"
  }
end

The “Then there should be a resource "Exec[#{arg1}]”“ business is another step definition, you can find it in features/steps/puppet.rb. Cucumber-puppet allows us to chain them together so that we can specify high-level things in the feature, but still get good test coverage.

And run the test:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

All greens!

2 scenarios (2 passed)
9 steps (9 passed)
0m0.745s

Working with manifests

Puppet-cucumber can instantiate nodes of class, but it can also read a site.pp file.

The features/support/hooks.rb file is where most of this is configured.

before do
  # adjust local configuration like this
  @puppetcfg['confdir'] = File.join(File.dirname(__FILE__), '..')
  @puppetcfg['manifest'] = File.join(@puppetcfg['confdir'], 'manifests', 'site.pp')
  @puppetcfg['modulepath']  = File.join(@puppetcfg['confdir'],  '..', '..')

  # adjust facts like this
  @facts["architecture"] = "i386"
  @facts["lsbdistcodename"] = "natty"
  @facts["lsbdistdescription"] = "Ubuntu 11.04"
  @facts["lsbdistrelease"] = "11.04"
  @facts["lsbmajdistrelease"] = "11"
end

Here we have a lot of control over cucumber-puppet’s behavior. We specify a puppet-confdir, a manifest, a modulepath, and facts. Since we are declaring a puppet-confdir, we can place a puppet.conf file there and configure almost any facet of the master or slave.

Since we are sourcing features/manifests/site.pp (non-standard—we made this up, and if you have a better idea please let us know), we can define nodes there. Add this node to the end of that file:

node "example" {
  include apt
}

And add the following to your apt_example_init.feature

    Scenario: Basic class, sourced from site.pp, sanity check
    Given a node named "example"
    When I compile the catalog
    Then compilation should succeed
    And all resource dependencies should resolve
    And the exec named "apt_update" should be run

And run the test:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

All greens!

3 scenarios (3 passed)
14 steps (14 passed)
0m0.921s

Rspec-puppet Mini Tutorial on Use

What are we testing?

The point of this very quick example is to test the apt class to see if it will create a file resource “sources.list”.

Create a very basic test

Rspec-puppet runs on Ruby files. The standard rspec-puppet layout has two spec subdirectories, defines and classes. Tests that focus on defines go in the define directory while tests that focus on classes go in the classes directory. The naming scheme is to use the name of the resource being tested, replacing :: with _ and ending with _spec.rb. So if we were testing the main apt class, we would call it apt_spec.rb and it would be placed in the spec/classes directory.

Let's create a file named apt_init_spec.rb as we have already defined an apt_spec.rb. (The absolute path, for clarity, is: ~/puppetforge/apt/spec/classes/apt_init_spec.rb)

require 'spec_helper'

describe 'apt', :type => :class do
  let(:title) { 'init' }

  describe "Apt class with no parameters, basic test" do
    let(:params) { { } }

    it { should create_class("apt")\
      .with_disable_keys(false)\
      .with_always_apt_update(false) }
  end
end

Run this command with:

$ rspec --color --format documentation spec/classes/apt_init_spec.rb

If all went well it should have read the second describe back, followed by ‘should create Class[“apt”]’ to you in green, indicating success. This is a very basic rspec test and it serves mostly to demonstrate that the catalog compiles and that the desired class is created. It should look like the following:

apt
  Apt class with no parameters, basic test
    should create Class["apt"]

Finished in 0.29773 seconds
1 example, 0 failures

Now let's look at the test line by line. The first line, require ‘spec_helper’ loads a file that will hold our default module path. It is explained further a little later. The next line defines our test. We are testing the ‘apt’ class and it is of type :class.

After each describe statement we can use let statements to define aspects of the class or define. For example, we choose the title for our class to be init. The next describe is a block to hold a whole test on the class. The describe takes a parameter which is the description for the test, between ‘describe’ and ‘do’. In this example, we have used a let for an empty set of params. If we wanted to pass params to the apt class we could also use ‘let(:params) { { :disable_keys => true } }’.

After the let, we use an ‘it’ block to actually describe a test. In this case we said that it should create a class using the create_class matcher. The create_class, and other create matchers can also take chained qualifiers like ‘with_disable_keys’ or ‘with_notify’. We can use this to verify parameters or properties of resources. Rspec-puppet will look for any parameter with the name of the string following ‘with’. So if you looked for .with_garbage, it would look for a parameter or property called ‘garbage’.

Add another test

Add the following to your apt_init.spec, below the first ‘it’ block. This will add a further test to the apt class we started working on.

    it { should create_exec("apt_update")\
      .with_subscribe(['File[sources.list]','File[sources.list.d]'])\
      .with_refreshonly(true)}

And run the test:

$ rspec --color --format documentation spec/classes/apt_init_spec.rb

All greens!

apt
  Apt class with no parameters, basic test
    should create Class["apt"]
    should create Exec["apt_update"]

Finished in 0.36027 seconds
2 example, 0 failures

You can configure rspec-puppet, assigning a module_path to work from. That is why we require ‘spec_helper’ in our test. ‘spec_helper’ should have contents similar to the following (this assumes the directory structure explained above):

require 'rspec-puppet'

RSpec.configure do |c|
    c.module_path = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
end

Going Further

Big Picture

The endgame for this project is to have continuous integration in place for modules on the Puppet Forge. We hope we have provided a decent framework for module developers to provide unit tests for their modules, but the project can only succeed with the involvement of the community. We can provide infrastructure and leadership, but by working in an open environment, we hope to test modules and reinforce standard procedures for implementing tools such as Jenkins, Hudson, and Buildbot with input and support from the community to optimize the testing process.

Technical Details

OPEN SOURCE

We have been working on two open source projects, rspec-puppet and cucumber-pupppet. Development on both projects is ongoing at our forks on github. There is an rspec-puppet fork at https://github.com/mlitteken/rspec-puppet/ and a cucumber-puppet fork at https://github.com/mlitteken/cucumber-puppet/. As of this post, all of our pull requests against the original repositories have been merged into the trunks.

WHAT WE CAN DO, BUT UGLY

On the cucumber-puppet front:

  • We can’t test a define without invoking a manifest.
  • We can specify facts, but only by including a yaml file or modifying hooks file which will be global for all tests.

On the rspec-puppet front:

  • If a test requires both a class and a define: we have to pass a “pre_condition” which is a hand coded Puppet manifest snippet, hard coded into the rspec test.
  • If we want to add facts: We pass a hash of facts for each test which is either global to all tests or hard coded for individual tests.

WHAT WE CAN’T DO, PERIOD.

On the cucumber-puppet front:

  • We can’t test Puppet 2.7.x.
  • We can’t access the scope object or any variables in scope.
  • We haven’t even begun to hook it into CI tools.
  • We can’t bulk test a ‘fact grid’ without hand-coding different facts sets.

On the rspec-puppet front:

  • We can’t access the scope object or any variables in scope.
  • We haven’t even begun to hook it into CI tools.
  • We can’t bulk test a ‘fact grid’ without hand-coding different facts sets.

This post was written by two of our development interns, Matthaus Litteken and Spencer Krum. Feel free to contact them with any questions, comments, or professions of undying love.

Share via:
Posted in:

Good to see how far we came from back then. rspec-puppet now exposes the scope, there is rspec-puppet-facts to access a good number of predefined sets of facts from real systems, and there are plugins for jenkins/rspec to parse and expose rspec output as results.

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.