Lockless Code Deployment in Puppet Enterprise & Continuous Delivery for PE
Puppet has introduced a new mode for deploying code in Puppet Enterprise that does not require stopping catalog compilation: lockless code deployments. This increases the number of catalogs that can be compiled per day.
Table of Contents
- Code Deployments in Puppet Enterprise and Continuous Delivery for Puppet Enterprise
- How Do Lockless Code Deployments Work?
- What Are the Benefits of Lockless Code Deployments?
- How to Enable Lockless Code Deployments
- Try Lockless Code Deployments
Code Deployments in Puppet Enterprise and Continuous Delivery for Puppet Enterprise
Your Puppet Enterprise (PE) installation’s primary job is to compile catalogs and send them to agents to be enforced. When you deploy code, all catalog compilation stops and waits for the code deployment to complete. This impacts performance in any installation, and the impact escalates with more frequent code deployments.
Code deployments usually increase when you use Continuous Delivery for Puppet Enterprise, making the impacts of stopped catalog compilations even more apparent.
How Do Lockless Code Deployments Work?
Lockless code deployments work by deploying multiple versions of your Puppet code and using the most recent version when a catalog compilation request comes in. Meanwhile, existing catalog compilations continue using the version of code they started with.
Find out more about how to enable lockless code deploys in Puppet documentation.
What Are the Benefits of Lockless Code Deployments?
If you’re already tracking your PE metrics, you will see the following positive changes in your metrics:
- Reduction in average JRuby wait time
- Increase in average number of free JRubies
- Elimination of number of fetch/commits with lock wait/hold
- Reduction in file sync timing from tens of seconds to almost zero seconds
- Smoothing effect on most metric graphs because there is no longer a lock followed by a burst of catalog compilations
Finally, you may see resolution to transient issues you didn’t realize had been caused by code deployments.
How to Enable Lockless Code Deployments
Please note that you will need to be using Puppet Enterprise version 2019.8.7 or later in order to use lockless code deployments.
You can enable lockless code deployments to deploy code without blocking requests to Puppet Server. (When deploying code, Puppet Server blocks requests, including catalog compilation, until file sync is done updating the Puppet code directory. ) With lockless code deployments, the file sync client updates code into versioned code directories rather than blocking requests and overwriting the live code directory.
You can choose to enable lockless code deployments on compilers individually or all at once. We recommend testing one compiler before enabling lockless code deployments on all Puppet Server instances. Then, you can enable it across the primary server, the replica, and all your compilers.
Enable Lockless Code Deploys on One Compiler
First, let’s walk through how to enable lockless code deployments on a single Puppet Server instance. Again, we highly recommend you test this on one compiler before enabling it across all instances!
In the compiler's node-specific Hiera file, set puppet_enterprise::profile::master::versioned_deploys to true.
Commit changes.
Run Puppet on the compiler twice: puppet agent -t; puppet agent -t
On your primary server, run: puppet code deploy --all --wait
Note: You must deploy all environments (with --all) to avoid errors.
It may take a decent amount of time (from a few hours to a day) for the compiler to exist with lockless code deployments until you are ready to enable them elsewhere. The puppet_metrics_collector module, enabled by default, collects data about Puppet Server that you can use to monitor the impact on the compiler after enabling lockless code deployments.
Once you're satisfied with the outcome, you are now ready to enable lockless code deployments on all Puppet Server instances. You can either repeat the steps above to enable lockless code deployments individually on all compilers, the primary server, and the replica, or you can enable lockless code deployments on all Puppet Server instances at once. However, as we’ve mentioned, you should only do the latter once you have tested this.
Enable Lockless Code Deployments on All Puppet Server Instances
To enable lockless code deployments globally – meaning on the primary server, the replica, and all compilers – you need to set the puppet_enterprise::profile::master::versioned_deploys parameter to true.
Remember: Enable lockless code deployments on one compiler as a test before enabling them across all Puppet Server instances. See above for the how-to.
We recommend using Hiera to enable lockless code deployments globally, but you can also enable this setting in pe.conf or the PE console (as configuration data).
To use Hiera to enable lockless code deploys:
Open your default Hiera .yaml file in a text editor. (For more information about Hiera data files, including file paths, refer to Configure settings with Hiera.)
Add the versioned_deploys parameter and set it to true. For example: puppet_enterprise::profile::master::versioned_deploys: true
Save the file and run Puppet on your primary server and all compilers twice: puppet agent -t; puppet agent -t
On your primary server, run: puppet code deploy --all --wait
Note: You must deploy all environments (with --all) to avoid errors.
To use the PE console to enable lockless code deploys:
Click Node groups and select the PE Infrastructure node group.
On the Configuration data tab, specify the puppet_enterprise::profile::master class, and select the versioned_deploys parameter.
Set the value to true, if the default value is not true.
Commit changes.
Run Puppet on your primary server and all compilers twice: puppet agent -t; puppet agent -t
On your primary server, run: puppet code deploy --all
Note: You must deploy all environments (with --all) to avoid errors. You can use the --wait flag to return results after all deployments finish, but this might hit timeout limits in large installations. Alternatively, you can check puppetserver.log to confirm that there were no errors during the code deployments.
Try Lockless Code Deployments
As you can see, the benefits of lockless code deployments are mighty – and they might even clear up transient issues. You can easily enable lockless code deployments on a single compiler, watch the metrics, and then enable it on the rest of your servers. Try it out and let us know how it goes.
Nick Walker is a Senior Product Manager at Puppet.
Learn more
- Read more about how Puppet Code Manager improvements reduce deployment time and effort.