published on 2 September 2013

Puppet Gatling is a Jenkins-CI plugin that post-processes Gatling simulation data to generate useful reports for load-testing Puppet. Using this tool, users are able to discover a clear difference in performance between various versions of open source Puppet and Puppet Enterprise.

Load Testing

Continuous Delivery:
What It Is & How to Get Started

Free Ebook: Continuous Delivery: What It Is and How to Get Started

» Smoother deployments.

» Happier teams.

» A more agile business.


The concept of load testing is pretty simple. Mostly it involves coming up with a few different ways to stress-test the code to understand how robust a given project is. This becomes very important when trying to show customers improvements.

One of my major projects as a summer intern at Puppet Labs was to find a simple way to show developers, marketing and customers that each version of open source Puppet and Puppet Enterprise improves on the prior version.

How did we accomplish this? Well there are a few main tools that we used, along with a little plugin development.

Jenkins CI

Logo for Jenkins continuous integration software

Jenkins CI (or just Jenkins) is an open source application that helps you execute repeated jobs like running different unit or acceptance tests, building packages, or other load testing. This style of development is known as Continuous Integration. Organizations will have either public or private Jenkins instances, or sometimes both. Some public examples include Apache, jruby, and even open source Puppet! Using Jenkins for continuous integration helps improve quality assurance and gives developers a way to keep a continuous check on the state of their application through development.

Jenkins also provides a huge variety of plugins to choose from to improve your experience. Because Jenkins is open source, it is pretty easy to write your own plugin too!

Gatling Project

Logo for Gatling load testing software

The Gatling Project has provided a plugin that you can hook into Jenkins which keeps track of different simulation information and a detailed view of each build that runs on Jenkins. By simply adding the plugin to a Jenkins job, it will display a few different trend graphs across builds including:

  • Mean response time trend
  • 95th percentiles response time trend
  • Percentage of failed requests

These graphs provide some great information about the job, however understandably so do not account for how Puppet runs work between a master and its agents. This means that the data is calculated incorrectly. Thus the need for a plugin extension, Puppet-Gatling.

Implementation of Puppet-Gatling

The main tools used in this process beyond Jenkins and Gatling include Maven with Java for plugin development. Maven proves to be a useful tool while developing Jenkins plugins for the simple reason that when you are ready to test, running the plugin through Maven will generate a new Jenkins instance. Without this feature, you would have to package your plugin every time and deploy it on a live Jenkins server.

Image of Jenkins console showing Puppet with Gatling

The src folder in Puppet-Gatling contains three main folders: java, resources, and webapp. The java folder is what you would expect: This is where all the java code for the project lives. The resources folder is for all of the view/GUI files that manipulate the Jenkins web console. These files are known as .jelly files. The webapp directory contains the plugin icon that will be displayed on a given Jenkins build.

The plugin itself is broken down into a few main classes that are important to Jenkins.

PuppetGatlingBuildAction.java

Image of sidebar in Jenkins This is what controls the code behind specific build pages on the web console. This page can be reached by clicking on some of the builds on the left-hand side of the main page for a job, as seen below. When visiting the specific build page for a given run, you will find some OS data that the project gatling-puppet-load-test gathered during the simulation (discussed below under the heading Additional Open Source Contributions), along with a breakdown of the data calculated by Puppet-Gatling. All the data you see on the build page is generated initially through the index.jelly file in the resources folder. Some of the code in index.jelly calls the related java file here as well - that is, to obtain some of the data collected and stored within the plugin. Think of jelly files like View in an MVC framework.

PuppetGatlingProjectAction.java

This is what controls the plugin main page for each Jenkins job. The icon on the left of the job dashboard provides a link to the project page. This page shows all the different trend graphs that Puppet-Gatling calculates for each run. This page also provides links to builds at the bottom, although you can also click on any data point within the graph to load the build page, too. This data point feature was added with my plugin, and was later merged into the official plugin.

Image of project view in Jenkins dashboard

PuppetGatlingPublisher.java

Image of post-build dialog box This is the main brains behind the project. The publisher part of that file name is important. For plugins, publisher tells Jenkins that it should be a post build action. Simply, this means that the plugin should run after everything else in the job has executed. This is important because if we run the plugin too soon, the Gatling data will not have been generated yet. This file is in charge of gathering all the data generated by gatling-puppet-load-test and the Gatling plugin, then crunching the numbers and providing the user with a pretty trend graph that has data related to Puppet runs.

Other java files in the project include: a class that stores the Simulation configuration; some data about the simulation; a report about the data collected from Facter for the simulation; and some other classes that help run the display of different trend graphs.

Testing Setup

A simple example job configuration is shown below. It will install Puppet Enterprise 2.8, and run two simulations. Each simulation has a unique id, and each simulation can have a different number of nodes to test against, along with the number of times it executes the node config. This style of configruation allows for ease in configuring a given Jenkins job. Before this, you had to go through a complicated process of setting env variables and adding a lot of shell scripts to execute. Thanks to the gatling-puppet-load-test project, you can now configure the job like this json below.


{
  "master": "master.localdomain",
  "sbtpath": "/usr/local/Cellar/sbt/0.12.4/libexec/sbt-launch.jar",
  "steps": [
    {"install": "pe28"},
    {"simulate": {
        "id": "PE28_bigcatalog_plus_vanilla",
        "puppet_version": "pe28",
        "scenario": {
          "run_description": "PE28BigCatalogPlusVanillaCent5",
          "nodes": [
            {
              "node_config": "pe28_bigcatalog_cent5.json",
              "num_instances": 3,
              "ramp_up_duration_seconds": 1,
              "num_repetitions": 3
            },
            {
              "node_config": "pe28_vanilla_cent5.json",
              "num_instances": 1,
              "ramp_up_duration_seconds": 1,
              "num_repetitions": 1
            }
          ]
        }
      }
    },
    {"simulate": {
        "id": "PE28_vanilla_5",
        "puppet_version": "pe28",
        "scenario": {
          "run_description": "PE28VanillaCent5, 1 instance, 1 repetition",
          "nodes": [
            {
              "node_config": "pe28_vanilla_cent5.json",
              "num_instances": 1,
              "ramp_up_duration_seconds": 1,
              "num_repetitions": 1
            }
          ]
        }
      }
    }
  ]
}

Our end product will test the same simulation against two different versions of Puppet. This can be done by having a new simulation step where you add an install step before the simulation. It will then uninstall whichever version of Puppet is present, add the new installation, and go about the simulation.

Gatling Puppet Load Test

This project is what allows for easy configuration in Jenkins. When the job configuration is in this json format, the gatling-puppet-load-test project is able to parse the simulation configuration and set up the Jenkins job for it to run with the given description. My part in this project was fairly small, and I contributed only a couple of features to the project itself.

One important feature of the Puppet-Gatling plugin is the ability to display different information about the OS and the simulation to the user. For that to happen, there needed to be a step in the setup of gatling-puppet-load-test using Facter to gather data. This was all done in ruby. Once the data required was collected, the setup step would save that data to the Jenkins workspace for the plugin to use later.

Another small contribution I added was the ability to make comments in the json configuration. This is important when you don’t want to install Puppet Enterprise on every job, or if there’s any other simulation step that you want to leave out. Otherwise you would have to move it out of the json configuration and save it as a comment in the execute shell portion of the Jenkins job configuration. This can become messy and annoying, especially when there are a lot of steps that you don’t need more than once.

Additional Open Source Contributions

As mentioned above, a feature of the Puppet-Gatling plugin was merged into the working source for the official Gatling Jenkins plugin. The main idea behind it was the ability to click on some of the graph data points and be taken directly to a build screen that Gatling generates. Without this feature, you had to navigate to the bottom and look through all of the builds in list order. This becomes a problem when you are trying to remember the specific build number and simulation id that you want to look at from the graph. Now you don’t have to: Just click on the data point, and it will open a new tab and take you there!

Final product

The plugin now lives on Github and will be maintained by Puppet Labs. However, it’s open source! So pull requests and contributions are welcome.

Learn More

Brian Cain was an engineering intern this summer with the Puppet Enterprise team. He is currently working on his masters degree in computer science at Kansas State University. In his free time, Brian enjoys sharing code on Github and his random thoughts on Twitter as @brian_cain.

Share via:

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.