Running Multiple Versions of Puppet with Puppet Enterprise, or: How I Learned to Stop Worrying and Love the Upgrade

Upgrading Puppet can be a long and involved process, particularly when skipping major versions. New language features are a included in each Puppet major version release. These changes, however, mean that upgrading Puppet is not a simple matter of installing a new package. The Puppet master should be upgraded to the new version first, then tested, then all of the agents upgraded and tested. Edge cases and problems may arise during this process because of the changes to the language and architecture. In an ideal world, you could safely transition to the new version (and mitigate impact of an upgrade) while leaving the existing Puppet deployment untouched. If it isn’t broken, why fix it, right? Fortunately, Puppet Enterprise provides a convenient and easy solution. This post describes how to use Puppet Enterprise to easily install two versions of the Puppet Agent on all of your systems, allowing you to try out upgrades without losing the functionality of your current deployment. Keep in mind that Puppet Enterprise is free for a small number of nodes; it provides the ability to try out new versions of Puppet without affecting your existing version. Puppet 2.6 introduces a number of new features that make writing and maintaining configuration code much easier than previous versions. Just a handful of compelling reasons to upgrade are the new REST API, Parameterized classes, and Run stages, resource chains in Puppet 2.6.

Traditional Puppet Upgrade Process

The process of upgrading Puppet involves reviewing the release notes for the new version comparing them to the version of Puppet you’re currently running. When skipping multiple versions (like moving from 0.24.8 to 2.6.5) this process can be long and involved. Backwards compatibility between Puppet versions is a goal of Puppet Labs, but is not guaranteed—and, as with any software, there comes a point where continuing to support much older versions is detrimental to progress. (The world quickly discovered just how much effort goes into backwards compatibility when the Windows 2000 source was leaked in 2004.) As a result, there are potential issues when older versions of Puppet Agents are connected to newer versions of the Puppet Master. The high level steps involved in a successful upgrade are:

1. Install the new version of Puppet on a separate Puppet Master. You’ll usually want to spin up a new virtual machine for this purpose.
2. Copy all of your Manifests and Modules to the new Puppet Master. This isn’t such a big deal with git or subversion.
3. Test and validate your existing Puppet Agent nodes against the new Puppet Master. This step is where things get a little messy.
4. Configure all of the existing nodes to cut over to the new Puppet Master after acceptance testing has passed.
5. Deploy the new version of the Puppet Agent on a select few testing nodes. Puppet makes this process quite pleasant.
6. Run through acceptance testing on these testing nodes.
7. When acceptable, cut the entire infrastructure over to the new version of the Puppet Agent.

This process often takes a team of experienced engineers several weeks to complete. With great power comes great responsibility, and the entire configuration of your data center is potentially impacted by an upgrade of Puppet.

Multiple Versions with Puppet Enterprise: A Handy How-to Guide

In this section, I’ll describe how to preserve an existing Puppet 0.24.8 installation on an agent system while also installing 2.6.4. This significantly reduces the time and effort involved in the Puppet upgrade process (specifically the process of acceptance testing and cutting over Puppet versions). First, we have a an existing Puppet Agent system connected to a working Puppet Master. A spare system or virtual machine provides the new Puppet Master using Puppet Enterprise. This separation of the the new and old Puppet Master systems provides two unique Puppet infrastructures allowing you to prevent one from interfering with the other. Installation of Puppet Enterprise is quick and easy (as shown in our installation post). With the new Puppet Master, we’ve reduced the time and effort with Steps 1 and 2 of the traditional setup described in the previous section. Next, you should copy your existing manifests and modules into the Puppet Master’s module search path located at /etc/puppetlabs/puppet/modules and /etc/puppetlabs/puppet/manifests which differ from the standard Puppet paths of /etc/puppet/manifests and /etc/puppet/modules. If you keep your manifests and modules in a different file system directory, you may easily change the location by editing PE’s modulepath setting in /etc/puppetlabs/puppet/puppet.conf. Once your new Puppet Master has been configured, it’s time to install the second version on a client node using Puppet Enterprise. When prompted by the Puppet Enterprise agent installer, provide the hostname of the new Puppet Master system rather than your existing system. The second agent allows you to easily validate your existing Puppet modules and manifests using both 0.24.8 and 2.6.4. If your existing Puppet version is version 0.25.5 or older, you may need to adjust some of the configuration sections, as their names have changed. With an older agent, you simply need to copy /etc/puppetlabs/puppet/puppet.conf to /etc/puppetlabs/puppet_acceptance_testing/puppet.conf and then change the [agent] section to [puppetd] and [master] section to [puppetmasterd]. For example:

# mkdir /etc/puppetlabs/puppet_acceptance_testing
# cp /etc/puppetlabs/{puppet,puppet_acceptance_testing}/puppet.conf
# perl -pli.bak -e ‘s/[agent]/[puppetd]/’ 
 /etc/puppetlabs/puppet_acceptance_testing/puppet.conf

(Note, the single quotes in the Perl command is a single quote if you copy and paste). These three commands create the new directory, copy the PE puppet.conf file and translate the names of the Puppet Agent and Puppet Master configuration sections. If your existing installation of Puppet is version 2.6.0 or newer, you simply need to copy puppet.conf into etc/puppetlabs/puppet_acceptance_testing as the section names are already correct. Let’s see how this configuration file for the new Puppet Master helps identify issues when skipping major versions during an upgrade. First, we have both versions available.

[root@agent ~]# /usr/sbin/puppetd --version
0.24.8
[root@agent ~]# /opt/puppet/bin/puppet agent --version
2.6.4 (Puppet Enterprise 1.0)

We’re also able to run the old version of Puppet against our freshly minted Puppet 2.6 master:

[root@agent ~]# /usr/sbin/puppetd 
 --confdir /etc/puppetlabs/puppet_acceptance_testing 
 --test --noop
info: Retrieving plugins
err: /File[/var/opt/lib/pe-puppet/lib]:
Failed to generate additional resources during transaction:
Fileset paths must be fully qualified
info: Caching catalog at /var/opt/lib/pe-puppet/localconfig.yaml
notice: Starting catalog run
info: Sent transaction report in 0.22 seconds
notice: Finished catalog run in 0.25 seconds

Unfortunately we’re getting an error about synchronizing plugins, so there’s some behavior change between 0.24.8 and 2.6.4 we’ve identified as a potential issue. We can verify this is not an issue when connecting the Puppet 2.6.4 agent to the new Master:

[root@agent ~]# /opt/puppet/bin/puppet agent --test --noop
info: Retrieving plugin
info: Caching catalog for agent.puppetlabs.vm
info: Applying configuration version '1298062572'
notice: Finished catalog run in 0.01 seconds

No error is present. You’ll need to determine if this problem is a potential issue when upgrading all of your agent nodes. We are receiving a valid configuration catalog from the new 2.6.4 server using a 0.24.8 agent, so we’ve verified Step 5. With future versions of Puppet there are potential situations where old Puppet agent versions are not able to cope with the new configuration catalog format served by the master. Installing Puppet Enterprise concurrently with Puppet allows us to easily and quickly validate this is not the case. With both versions of the Puppet Agent installed, it’s now possible to validate steps 6 and 7, acceptance testing and cutting over to the new version of Puppet. If you encounter issues, you can immediately switch back to the existing Puppet version by simply calling /usr/sbin/puppetd.

Conclusion

Puppet Enterprise allows engineers to greatly simplify the process of testing and validating new versions of Puppet. Many members of the Puppet community install two versions of Puppet on the same system to run through this process. My previous strategy to address this problem worked well enough: to add the envpuppet script to the Puppet source tree. However, rather than having to write and maintain a hacktastic program, Puppet Enterprise provides an officially supported and easy to deploy suite of packages that greatly reduces the work involved with upgrading to new versions of Puppet. Try it out now at http://www.puppetlabs.com/.