Upgrading to Puppet Enterprise 3: Avoid the Gotchas
It’s time to upgrade! As Puppet Enterprise 2 nears its end of life, it's time to start thinking about an upgrade to Puppet Enterprise 3.
As a support engineer here at Puppet Labs, I've helped a fair number of customers deal with the bumps and bruises along the road to a successful upgrade to the 3-series of Puppet Enterprise.
To help you along the way, I've collected some of the most common "gotchas" and issues customers have run into while trying to upgrade, along with some useful solutions, workarounds, and mitigating steps.
NB: For the purposes of this blog post, we'll assume you're upgrading to Puppet Enterprise 3.3, our newest release.
If you've been running Puppet Enterprise for a long time, and haven't ever pruned your console database, now's the time!
An upgrade from Puppet Enterprise 2-series to 3-series involves a database migration from MySQL to PostgreSQL, and a large unpruned console database can severely extend the time it takes for an upgrade to complete. Under extreme circumstances, a very large database can result in an error during the upgrade process, and cause a failure.
We recommend pruning to 30 days of report history, but you can tweak that to your company's specific needs.
Find documentation on pruning the PE 2-series console database here: <http://docs.puppetlabs.com/pe/2.8/console_maintenance.html#cleaning-old-reports>
I also highly recommend my colleague Nick Walker's previous blog post about ongoing console maintenance, available here: <http://puppetlabs.com/blog/how-maintain-your-puppet-enterprise-console>
Module and Manifest Issues
Duplicate Module Names
One common problem we see is customers writing modules to manage functionality that's actually built into Puppet Enterprise 3, resulting in a module-name collision that can cause failed upgrades.
This can cause issues with Puppet runs performed during the upgrade, as the wrong module may potentially be used, due to naming collisions between your modules and the PE-included modules.
This issue was mostly resolved in Puppet Enterprise 3.2 and 3.3, via the namespacing of built-in modules, but it's worth double checking nonetheless. Here are the modules built into PE 3.3:
If you've got a module that shares a name with the list of modules above, you should at a minimum, refactor that module with a new name to avoid conflicts. If possible, you should refactor your code to use the Puppet-provided module.
In the case of
puppetlabs-apt for example, it may no longer be necessary to maintain your own apt module — the official apt module has rich functionality and will likely meet your needs.
Dynamic Variable Scoping
Dynamic scoping for variables was removed in Puppet 2.7, and completely removed in Puppet 3, the version of core Puppet on which the 3-series of Puppet Enterprise releases are built.
If you're using dynamic scoping, you'll see deprecation warnings like these in your puppet runs:
As the error states, the best reference on how to update your manifests can be found in our docs — <http://docs.puppetlabs.com/guides/scope_and_puppet.html> — so I won't reproduce that information here.
If you're not seeing those errors on your agents during Puppet runs, you're good to go!
Ruby 1.9.3 Language Changes
In the Puppet Enterprise 3-series, we upgraded from Ruby 1.8.7 to 1.9.3. This is a large change, and brings with it a host of performance improvements and other benefits.
There are a few changes in Ruby 1.9.3 that may affect the compatibility of your ERB templates and other custom ruby code (like custom facts, types and providers, and functions).
If you have incompatible ruby code in a custom fact, type, or provider, it may even prevent your upgrade from succeeding.
Strings No Longer Enumerable
In Ruby 1.8.7, strings were enumerable. This meant that you could iterate over a string, line by line, with no modification. In Ruby 1.9.3 this is no longer the case:
In Ruby 1.9.3, you can recreate that functionality with the
For other methods like
.inject, you'll need to create an enumerator object, which you can then call
map on. You can enumerate lines, characters, or bytes — but you'll probably want lines in most cases, if you're trying to mimic ruby 1.8.7 behavior:
The above changes may require you to refactor some of your existing ERB templates, custom facts, and any other places where you're writing pure Ruby code for use with Puppet Enterprise.
String Encoding Changes
Due to changes in the way Ruby 1.9.3 handles string encoding, we recommend avoiding the use of non-ASCII characters throughout your puppet code. We're working on improving the way mixed encodings are handled in Puppet, but for now sticking with ASCII characters is your best bet, as using non-ASCII characters may cause your puppet runs to fail under some circumstances.
More on the work being done to address this here: https://tickets.puppetlabs.com/browse/PUP-1031
Console Auth Disabled for AD/LDAP Authentication
For some older releases of PE, it was necessary to disable console auth in order to use AD or LDAP authentication in the console. This is no longer the case in either PE 2.8.x or PE 3-series releases, but the configuration lingers in many deployments.
If you attempt to upgrade with
console auth disabled, you may experience issues logging in to the console after your upgrade.
You can check on this by running
ls -l /etc/puppetlabs/httpd/console_apps.d/.
If you see files with the word
disabled appended to the end, you've disabled console auth.
We recommend re-enabling console auth prior to the upgrade, and working with local console credentials for the duration of the upgrade process. Then, once the upgrade is complete, follow the instructions to configure third-party authentication services.
You can re-enable console auth by running the following command on your console node prior to upgrading:
Take a Backup
This part gets its own section, because we can't stress it enough. Before you upgrade from Puppet Enterprise 2 to the 3-series, we strongly recommend that you take a complete backup of your entire Puppet Enterprise infrastructure.
If you're running a virtual infrastructure, or have the ability to snapshot your bare metal, we recommend snapshotting your PE infrastructure. That's your master if it's an all-in-one install, or your master and console if it's a split install.
You'll also want to back up the
console_inventory_service databases if you're using an external, self-hosted MySQL database that isn't covered by your snapshots.
If you're not able to snapshot your PE infrastructure, you'll want to back up:
/etc/puppetlabson your master and console.
/opt/puppet/share/puppet-dashboard/certson your console.
- The MySQL databases, as described above.
- You can use
mysqldumpto accomplish this. For example, running
mysqldump -uroot -p --all-databases > console.sqlon your all-in-one master would back up all PE databases.
- You can use
Puppet Enterprise 3.3 Environment Changes
The latest version of Puppet Enterprise is 3.3, and there have been a few issues specific to that release worth addressing here.
As you may be aware, Puppet Enterprise must always have
/opt/puppet/share/puppet/modules in the modulepath for all environments.
This will always be the case on new installations of Puppet Enterprise 3.3. However, if you upgrade and attempt to enable directory environments,
/opt/puppet/share/puppet/modules may be removed from your modulepath unless you take certain steps.
If you enable directory environments, you’ll want to, at a minimum, set
[main] to include
If you plan to use
environment.conf files in your directory environments to set modulepath, you’ll need to include
/opt/puppet/share/puppet/modules there as well.
Our recommendation would be to set
$basemodulepath = /opt/puppet/share/puppet/modules in
puppet.conf, and set modulepath in each environment’s
environment.conf file to
modulepath = modules:$basemodulepath.
This results in a final modulepath of:
That about covers the common upgrade issues we see in support tickets at Puppet Labs, but there are definitely other backwards incompatible changes to Puppet you'll want to look out for.
I'd recommend checking out the release notes for Puppet 3.0.0, paying special attention to changes marked BREAKING: http://docs.puppetlabs.com/puppet/3/reference/release_notes.html#puppet-300
Because Puppet Open Source is semantically versioned, we don't introduce backwards-incompatible changes in .y releases, as in x.y.z, so the changelog for Puppet 3.0.0 is where you'll find breaking changes.
- You can always check the Puppet Enterprise lifecycle here: http://puppetlabs.com/misc/puppet-enterprise-lifecycle
- Puppet Enterprise 3.3 was launched this week. For more on that: PE 3.3 Launch Announcement
- PE 3.3 Documentation