homeblogintegrate litmus acceptance tests into cd for pe

Integrate Litmus acceptance tests into CD for PE

This blog post shows how to combine Puppet open source testing tools PDK (Puppet Development Kit) and Litmus with Puppet Enterprise and Continuous Delivery for Puppet Enterprise (CD for PE). I will first show how to add Puppet Litmus acceptance tests to a Puppet module. Afterwards I will show how to integrate these acceptance tests with Continuous Delivery for Puppet Enterprise. This gives you automated pipelines to check our Puppet module code.

This blog post covers:

  • Using Puppet Litmus for acceptance tests
  • Converting existing modules to run Litmus acceptance tests
  • How to use Puppet Development Kit (PDK) for syntax validation and unit tests
  • Using Puppet Bolt to check what is installed in our acceptance test environment
  • How to integrate acceptance tests into Continuous Delivery for Puppet Enterprise to achieve automated module testing

Puppet module testing

There are three ways to test your Puppet module:

  • Syntax validation
  • Unit tests
  • Acceptance tests

Syntax validation

Syntax validation can be done within your IDE. When writing Puppet code, I would recommend using Visual Studio Code. You can install the Puppet extension in Visual Studio Code which includes features like syntax highlighting, linting, debugging, IntelliSense, and much more. These features can greatly help speed up the process of creating Puppet code/configurations. Another key benefit of using Visual Studio code when creating Puppet code is that you can easily integrate with the Puppet PDK meaning that all of your code creation and validation can be done within a single tool.

Puppet Development Kit (PDK)

PDK provides a complete module structure, templates for classes, defined types, tasks, and a testing infrastructure. You can validate and test your module against various operating systems and multiple Puppet versions. You can also convert existing modules to make them compatible with PDK. This allows you to use all of the creation, validation, and testing tools in PDK with the converted module.

To validate the syntax and coding guidelines compliance of your code, go to the top level directory of your module and run the command below. This will:

  • validate the metadata.json file
  • validate Puppet syntax
  • validate Puppet code style
  • validate Ruby code style

In Visual Studio Code you can open an internal terminal and run all the commands in this terminal without leaving the IDE.

Giving the --parallel parameter runs the validation tasks in parallel. If you add the -a parameter, PDK will try to make your code compliant to the coding guidelines (autocorrect). You will get information on errors PDK was able to fix and about what problems you have to resolve yourself.

Please refer to the PDK documentation for further information about PDK.

Unit tests

Unit tests are more costly than syntax validation and are not as costly as acceptance tests. In the case of a Puppet module, unit tests create a Puppet catalog from the module’s Puppet DSL. The unit tests inspect the catalog to prove that defined resources are present with the desired configuration. An example for a unit test is to check if a service is defined in the catalog and has the right attributes for ensure and enable. Passing a unit test shows that your code is working as expected. That’s why unit tests save time. You can check your code running unit tests before pushing it into a source code repository or deploy it on a server.

This post is not about writing unit tests but here you can find an example for unit tests. More detailed information is available in the Puppet rspec testing documentation. The unit tests are located in the spec/classes or spec/defines folders of a Puppet module.

I highly recommend writing unit tests to check whether your code is doing what you expect before starting development of acceptance tests.

PDK runs unit tests with the following command:

The --parallel parameter will run as many unit tests in parallel as possible on your machine. This depends mainly on CPU cores available. The -v parameter makes the output more verbose.

But unit tests do not apply a Puppet module onto a server. That’s where acceptance tests come into play.

Difference between unit and acceptance tests

Before we dig into acceptance testing, let’s discuss the differences between unit and acceptance testing.

The main difference between PDK unit tests and Litmus acceptance tests is that unit tests only inspect the catalog created from the module's Puppet DSL but cannot test changes to the managed systems or services. Also idempotency of your module code can’t be tested. That is what Litmus acceptance tests do. Litmus acceptance tests are running the Puppet module on real environments, and they change these systems and/or services by applying the Puppet module. And Litmus tests also test that your module code is idempotent.

Acceptance tests are therefore more costly than syntax validation or unit tests. The reason for this is that acceptance tests with Litmus are starting a test environment (e. g. Docker containers, virtual machines), and apply the Puppet module code to this test environment. Applying the Puppet module consumes more time than creating only a catalog and checking resources in this catalog, as there’s the real work to be done. Saying this I recommend to start with syntax validation, write and run unit tests, and if these two tests are passed, write and run acceptance tests.

Litmus acceptance tests

From the Litmus wiki:

Litmus is a tool that helps acceptance test Puppet modules. It facilitates testing a module against a variety of OSes, and deployment scenarios. This tool helps provision test platforms such as containers/images, install the Puppet agent, install a module and run tests with minimal effort.

Before you start

Please install the following tools on your computer:

The module you want to add Litmus acceptance tests to has to be compatible with Puppet Development Kit (PDK). To check if your module is compatible with PDK, please have a look into the metadata.json file. There should be an entry like pdk-version: together with the version of PDK the module is compatible with. If your locally installed PDK version is newer than the PDK version in the metadata.json file, please update your module running pdk update first. If your module is not yet compatible with PDK you can convert the module by running pdk convert from the top level folder of your module. Converting a module will add files and change some things so be sure to have committed all your changes or have a backup just in case … But PDK will show you which files would be changed and you have to accept these changes.

For this blog post Docker will be used to run containers for the acceptance tests. Make sure the user you use to run the acceptance tests has access to Docker, e. g. can run Docker ps or other Docker commands. If this is not the case, on Linux systems check if the user is in the Docker group.

Configuring Litmus acceptance tests

The Puppet Litmus project can be found on GitHub. To run Litmus acceptance tests we need to add and update some files in our Puppet module directory.

This blog post is based on my Puppet module sshkeymgmt and shows how to add Litmus acceptance tests.

Edit .fixtures.yaml

The .fixtures.yaml file loads all listed modules for unit and acceptance tests either from the Puppet Forge (forge_modules) or from public or private Git repositories. Add the following lines to your .fixtures.yaml file and make sure there are no duplicate entries in that file and you added all dependencies needed.

Your .fixtures.yaml file should look similar to this and should contain all further modules you want to use.

Create the provision.yaml file

The Puppet provision module allows provision and tear down of acceptance test systems. It is also responsible for creating a Bolt inventory according to the provisioned environment.

Currently four provisioners are available:

  • ABS (AlwaysBeScheduling)
  • Docker
  • Vagrant
  • Vmpooler (internal to Puppet) The provision.yaml file is located in the top level directory of your module and contains the configurations for your various acceptance test environments. The following example will show how to create a test environment with Docker and another with Vagrant, both with two target systems.

We will use the Docker environment for our acceptance tests.

Create spec_helper_acceptance.rb file

The spec_helper_acceptance.rb acceptance testing file is located in the spec folder. If this file is absent, there are no acceptance tests for the module. The code below loads the puppet_litmus gem and runs the Litmus configuration. It includes a require statement for a spec_helper_acceptance_local.rb file which will be loaded only if it exists. The spec_helper_acceptance_local.rb file can contain more helper functions to support your acceptance tests.

Create spec_helper_acceptance_local.rb file

The spec_helper_acceptance_local.rb file is also located in the spec folder. The file should contain local helper functions. The file below is a short example which you can extend to your needs. It contains a helper function test_sshkeys which applies Puppet code and tests a certain file to have an expected content. I added the test_sshkeys function to demonstrate for this blog post how to check a file to contain some expected content. The read_hosts_ssh_ports function shows how to read the auto-generated inventory file and the connect_by_ssh function does a ssh login to a host and runs the hostname command. The login test is a good example for the difference between unit and acceptance tests, as the unit test only checks for the ssh keys to be in the catalog, and the acceptance test does the login to check if the key is working.

The idempotent_apply function in the code below applies the given Puppet code twice to test if the code is idempotent.

Depending on your modules and the tests you need to run, you can add more helper functions to the spec_helper_acceptance_local.rb file.

Update Gemfile

Edit your Gemfile and add the following two lines in the group :development section.

The lines above load the puppet_litmus and the servespec gems. For one of our acceptance tests we will login into the acceptance test instances using ssh. For this the net-ssh, ed25519 and bcrypt_pbkdf gems are needed.

Update Rakefile

Edit your module's Rakefile and add the following line if it does not yet exist:

Add acceptance tests

To add acceptance tests, go into the spec directory of your module and create a folder acceptance. Acceptance tests will be located in the acceptance folder. Litmus acceptance tests are based on serverspec. Serverspec is a tool to test your server’s actual state. In our case we use Puppet to apply our module code on the test servers and check for the results with serverspec.

Below there's a simple code example for an acceptance test. This example code below defines a piece of Puppet code and stores it into the pp_ssh_dir_in_user_home variable. The Puppet code calls the class sshkeymgmt and passes a user configuration to the module. To run the test, the test_sshkeys function is called to check if the given file has the expected content. The second test uses the serverspec test describe file to check for the expected content of a file.

The file has to be named <module name>_spec.rb.

More detailed acceptance tests can be found here.

It is recommended to use serverspec for writing acceptance tests for Litmus.

You can use tests like this as well:

Running acceptance tests

Before we can run the acceptance tests we need to do some pre-work. First we need to install all necessary Ruby gems.

Install required gem files

Run the command above from the top level directory of your module. This will install all required Ruby gem files.

Start acceptance test environment

To start the two acceptance test docker containers from the provision.yaml file you created before, run the following command from the top level directory of your module:

This will start two Docker containers on your local machine. One CentOS 7 and one Ubuntu 18.04 container. During the startup of the containers Litmus will create an inventory.yaml file. This file can be used with Puppet Bolt to access the containers. This inventory.yaml file will be created dynamically and all further Litmus tasks, like installing the Puppet agent, will use this inventory file. Litmus uses Bolt under the hood to connect to the test instances.

To verify if both containers are up and running just type the following:

This should give you an output similar to this:

If you want to use Terraform or some other test instances it is possible to provide a handcrafted inventory file. With this handcrafted file you can use the Terraform plugin for Bolt or every other Bolt plugin. In this case this provision step is not necessary.

Install Puppet agent

First we need a Puppet agent in our acceptance test environment. The following command will install the Puppet agent into all configured acceptance test instances (see your inventory.yaml file).

You should see only one line as output: install_agent.

After the install task has finished you can validate the Puppet agent installation by running the following command:

The output of the above command should be similar to this:

Install Puppet module

Now install the Puppet module into the acceptance test instances using the command below:

This command should print only the following two lines of output:

If you want to verify the installed modules on each instance the following command will do that:

Run acceptance tests

Acceptance tests can be run in parallel on all defined acceptance test targets. This will give you less output about the tests but will speed up testing. In fact, you get only a summary of the passed tests or some error messages if acceptance tests fail. The command below will start the acceptance tests in parallel:

Sometimes you need some more detailed output about the tests and what happens. In this case you can run the acceptance tests node by node. The following example assumes that the docker container is available by ssh on localhost port 2222. Information about the running acceptance test instances and how to connect to them can be found in the inventory.yaml file.

--format d gives a more verbose output about the tests.

The output will be similar to this:

Shutdown the acceptance test environment

This command will shutdown the acceptance test environment:

After running the command above, the two docker containers should be shut down. To verify which docker containers are running, you can use the following command:

Now we have everything complete to integrate our acceptance tests into a CD for PE pipeline.

Workaround for Vagrant

Bolt uses the vagrant user to log into the Vagrant boxes and uses the run-as feature of Bolt to run commands with sudo as a different user, mostly root. The Puppet agent is installed into the /opt directory. This directory is not included into the secure_path configuration of sudo. The Puppet provision module contains a Bolt task provision::fix_secure_path to add the Puppet bin directory to the sudo secure_path.

CD for PE and Litmus acceptance tests

Continuous Delivery for Puppet Enterprise is a commercial tool within the Puppet platform. CD for PE needs Puppet Enterprise and gives you the ability to create pipelines to do whatever is needed in your environment to ship your Puppet code from a development environment to a production environment. And CD for PE can do impact analysis if you have Puppet Enterprise running. The impact analysis shows you the potential impact changes in Puppet code may have on your PE-managed infrastructure.

Continuous Delivery for Puppet Enterprise also gives you the ability to run syntax validation, unit tests, and acceptance tests for your modules whenever a pull request or a git push comes in. CD for PE integrates with GitHub,GitLab, Bitbucket, and Azure Devops and gives you visibility of pipeline runs within pull or merge requests.

CD for PE contains predefined jobs for several tasks. Examples of predefined jobs are: syntax validation of a module with PDK or running unit tests with PDK.

Next to jobs available out of the box, CD for PE allows you to create your own jobs with whatever content you want. For Litmus acceptance testing we need to create our own Job in CD for PE.

Job hardware servers are organized by capabilities within CD for PE. A capability is a tag that indicates what type of jobs can run on that job hardware server.

CD for PE jobs can either run on dedicated hardware agents or within docker containers. As we use Docker to run our acceptance tests we will go for a dedicated hardware agent in CD for PE as running docker within docker would make this blog post too complicated. In my test environment I add the capability "LITMUS" to such job hardware to bind my job steps to this particular job hardware.

Before you continue, please make sure you have a job hardware available to use with Litmus. This job hardware needs Puppet PDK installed. Further information on how to add job hardware can be found here.

The following steps have been done on CD for PE version 3.3.

Create an acceptance test job

Go to the Jobs page in CD for PE and click on New job. You will see a popup where you can define your new job.

cdpe create job part

The new job needs a name and a brief description of what the job is doing. Now put in the commands to run when the job executes as well as the commands after job success and if the job fails. As we use Docker, we have to take care that we clean up our acceptance test environment. That has to be done after a successful and an unsuccessful run. Otherwise our environment gets messed up with a lot of unused docker containers. The tear_down command uses the inventory.yaml file created by the provision task to tear down the environment.

The following code block contains the commands to make copying easy.

Please make sure that missing Gem files can be built on the machine running the Litmus acceptance tests. This means you have to install all tools needed to build gems with native extensions. This page gives a good overview on how to do this on CentOS 7.

The next picture is about the job hardware settings and the Docker configuration.

cdpe create job part

Since we want to run Docker containers for our acceptance tests, our job should not run in a Docker container.

The capabilities depend on what is available in your environment. I normally assign the capability LITMUS to all job hardware that can run Litmus acceptance tests.

Add your module to CD for PE

Go to the Modules page in CD for PE. Click on Add Module to add a new module.

cdpe select module

The following 5 steps need to be done to add a new module to CD for PE. After completing each step the next part will show up.

  1. Select the source code control system which is hosting your module
  2. Select the organization where your module is located
  3. Select the repository
  4. Select the branch you want to use
  5. Enter the module name

After finishing these steps click Add to save your module.

Setup a module test pipeline in CD for PE

On the CD for PE Modules page, click on the new modules added before. You now can add a default pipeline to your module. Select the branch you want to use and click on Add default pipeline to start adding the new pipeline.

cdpe add default pipeline

The new default pipeline contains a Code Validation stage. This stage contains jobs for running code validation and unit tests. We do not need the Deployment stage at the moment and this stage can be deleted.

cdpe default pipeline

After confirming to delete the Deployment stage, add a new stage for our acceptance tests. Click on Add stage. You will see a popup window Add new stage.

cdpe configure new stage

Then create a new stage do the following:

  1. Enter a name for the stage
  2. Select Jobs in the Add to this stage dropdown
  3. Select the acceptance test job from the job list
  4. Click add stage

After the new stage is created, you’ll see there is an auto promote step between the code validation and acceptance test stage. If you tick auto promote, the acceptance test stage will run without manual interaction. You can select a condition when the stage should run. For our test set it to All succeeded. This will run acceptance tests only if syntax validation and unit tests finished without errors.

Run a test

There are two ways to test our new pipeline. Firstly, whenever you commit and push a change to your Git repository, the CD for PE web hook will fire and start the pipeline. Secondly, you can also trigger the pipeline manually.

cdpe trigger pipeline

After clicking on Trigger Pipeline the pipeline starts and a new event will show up on top of the events table. Clicking on an event will open the event and show the jobs configured. Clicking on the jobs shows the job logs.

Now we have configured a new pipeline in CD for PE checking the syntax of our module, running unit and acceptance tests.


With Puppet there are open source tools to help with testing your module code. PDK supports syntax validation and unit testing. These tests should run on your local machine before pushing your code to a Git repository. Litmus supports acceptance tests. Since syntax validation is the fastest test, you should always run this test first. Unit tests are not as costly as acceptance tests. That's the reason for running unit tests after the syntax validation. And last but not least, run the acceptance tests.

Combining PDK and Litmus with pipelines in CD for PE brings additional value, like clean and well-tested code to your Puppet module development. And you can deliver the tested code into your Puppet environments automatically. Adding impact analysis in CD for PE can help you to estimate what impact a change will have on your environments.

Example modules on GitHub and further reading

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.