To upgrade, or not to upgrade, that is the question
Managing complex IT systems is always a delicate balancing act for sysadmins — between getting the latest updates to take advantage of new features and bug fixes, as well as security updates, and keeping a stable system running without making any unnecessary changes. Add to this the often complex and time consuming change control procedures in place at many enterprises, and what develops is a situation where as long as there are no pressing problems, it is easier to leave production systems running the same versions of critical software until an upgrade is forced on the admins.
At Puppet, we recognize that this situation likely also applies to the Puppet Enterprise infrastructure. Upgrading was seen as an unnecessary upheaval, and with the historically rapid release cadence, upgrading to the latest version of PE was not always high on the list of priorities for busy sysadmins. In order to assist sysadmins in this position, we have made changes to our release schedule, moving from quarterly releases to twice yearly releases, and adopting a Long Term Support (LTS) release model where a specific release can be fully supported for up to two years. This allows organizations to pick either the LTS release version where stability is paramount, or the latest version where new features are released first where the value of these outweighs any potential risks. This is outlined on the Puppet Enterprise support lifecycle page for better planning visibility.
However, we would like to address this area a little further, both in terms of why upgrading is important, and how the upgrade process can be made more successful with minimized risk.
"There are more things in heaven and Earth, Horatio, than are included in older versions of PE."
Each release of Puppet Enterprise is a point in time, and as time moves on, so does development of the suite of products that make up PE. Across major releases (for example from PE 2017.2 to PE 2017.3) new features will be added, existing features may be significantly improved or changed, and deprecated features will be removed. If these new features will not be used in your deployment, this does not make a compelling case for upgrading. However, it is always a good idea to become familiar with the new features, perhaps by spinning up a test system, as you may find that you actually do have a compelling use case for a new feature.
"A King of bugs and patches"
Software has bugs, and bugs need patches. When Puppet fixes bugs, they are released as part of a Z release (with an incremented final digit like 2017.3.1). Whether you are on the LTS release (currently PE 2016.4) or on the latest release, we would always urge you to upgrade to the latest Z release to ensure you benefit from all the latest bug fixes and enhancements. As part of the support process for PE, you may be asked to upgrade to resolve a specific issue you may be experiencing, or to allow you to use PE to customize aspects of its operation to suit your requirements better.
In any case, upgrading to the latest Z release within the same version of PE will be a low-risk operation often with very significant benefits, and we would encourage all our PE customers to upgrade to the latest Z release as a matter of course. And because we believe in dogfooding, Puppet’s ops team is encouraged to upgrade to Release Candidate versions as they are released internally, so you can be confident that they have been run on production systems before you take the risks.
"Though upgrading be madness, yet there is method in't"
Upgrading any software always carries an element of risk, and at Puppet we want to ensure that the risks associated either with a Z release upgrade or a more major upgrade remain as low as possible. A well prepared, methodical approach is crucial.
To this end, we would always recommend following the steps below:
- Read the release notes, including known issues, for the target version.
- Test the upgrade on a staging system first. Testing an upgrade on a test system, where issues would not cause any production problems is a valuable way of ensuring production upgrades do not have any unexpected problems. It allows you to test your runbook or method of procedure (MoP) documents, and allows you to catch and correct any issues before the upgrade is done on live production platforms.
- Take backups of all infrastructure nodes. While this may seem obvious, sometimes people forget and we always recommend taking a full backup of the PE infrastructure nodes before commencing any upgrade, either by using a snapshot in the case of the infrastructure running on VMs, or by taking a backup of all aspects of the configuration including config and data directories (or even a full disk backup) in the case that it is not. In a worst case scenario, having a backup allows the upgrade to be backed out, simply by restoring a snapshot or backup. This reduces the pressure on the engineers performing the upgrade, knowing that any issues need not be catastrophic.
- Let Puppet Support know an upgrade is planned. While Puppet Support does not carry out upgrades on customer systems, or join live upgrade sessions, having insight into when an upgrade is planned and having some of the important details ahead of time will allow us to have better situational awareness in the event that you do encounter an issue and need to raise a support ticket. While we do not validate runbooks or MoPs, having them available will streamline the troubleshooting process, as we know exactly what was done and in what order.
- Document each step as it is applied, including screen output. Knowing at what stage an issue occurred, and the exact error messages produced, allows Puppet Support to focus in on specific areas of concern, and improve and streamline the troubleshooting process.
"I'll upgrade it, though it blast me"
Upgrading software is risky business, but we would strongly recommend that in the case of your Puppet Enterprise infrastructure, you make every effort to upgrade it to the latest version, whether on the LTS or current release. With some careful planning, you too can benefit from the latest Puppet can offer, and stay on the shortest path to better software.
With sincerest apologies to William Shakespeare for mangling his words from Hamlet.
Stefan Pijnappels is a senior support engineer at Puppet.
- To see what’s new in the latest version of Puppet Enterprise, read through the release notes.