PE release notes

These are the enhancements and resolved issues in this version of Puppet Enterprise (PE).

For security and vulnerability announcements, see Security: Puppet's Vulnerability Submission Process.

PE 2023.8.0

Released August 2024

Important: Puppet Enterprise (PE) 2023 is our current PE LTS release stream. The previous LTS, PE 2021.7, is in overlap support until 28th February, 2025.

For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z are encouraged to upgrade to either 2021.7 or 2023.

Enhancements

Default to find reports generated within the last 30 minutes on the Events screen in the PE console
In order to make the page load faster and be more efficient, the Events screen in the PE console has changed the default period from Events from the last run to Events in the last 30 minutes.
Lockless code deploys enabled by default
Lockless code deploys is now enabled by default. The default of locking all compilation processes to complete each deployment of puppet code is no longer enabled. As a requirement of this release, the codedir is changed from /etc/puppetlabs/code to /etc/puppetlabs/puppetserver/code.
Lockless code deploys defaults updated
The defaults for the Lockless Code Deploys feature of Code Manager (which since version 2023.7 is the default way to deploy code), have been updated with a faster method of deploying each environment and the capacity to deploy 2 (configurable) environments at a time. See Configure Code Manager for puppet_enterprise::master::file_sync::copy_method and puppet_enterprise::master::file_sync::versioned_sync_pool respectively.
JRuby spawning initialization improvement
Puppet Server now initializes one JRuby instance and once it is initialized, further instances are initialized concurrently, up to a configurable max level of concurrency. This level of concurrency is configurable via class parameters, data, or the Hiera value of puppet_enterprise::master::puppetserver::jruby_puppet_instance_creation_concurrency.
Experimental setting to potentially improve Puppet Server startup time
Customers may now enable an experimental setting that could improve Puppet Server startup time by speeding up the per-JRuby instance creation time. This is controlled through the new parameter: puppet_enterprise::master::puppetserver::settings_catalog.
Usage of find and chown in lockless Puppet code improved
A slow and I/O intensive operation in compiler catalogs (codedirs chown) is now optional and may be disabled with the puppet_enterprise::master::file_sync::chown_code_to_pe_puppet parameter.
Code management parameter deprecations and new parameter improvements
The following parameters are deprecated:
  • puppet_enterprise::master::code_manager::git_settings
  • puppet_enterprise::master::code_manager::private_key
  • puppet_enterprise::master::code_manager::forge_settings
Instead of providing one large JSON object to the git_settings and forge_settings parameter, multiple simpler parameters have replaced the deprecated parameters and the replacement parameters are also on a new class:
  • puppet_enterprise::master::code_management
The replacement parameters for the git_settings parameter are:
  • puppet_enterprise::master::code_management::git_provider
  • puppet_enterprise::master::code_management::git_private_key
  • puppet_enterprise::master::code_management::git_default_ref
  • puppet_enterprise::master::code_management::git_proxy
  • puppet_enterprise::master::code_management::git_oauth_token
  • puppet_enterprise::master::code_management::git_repositories
The replacement parameters for the forge_settings parameter are:
  • puppet_enterprise::master::code_management::forge_proxy
  • puppet_enterprise::master::code_management::forge_baseurl
  • puppet_enterprise::master::code_management::forge_authorization_token
For further information see Customize Code Manager configuration in Hiera.
Install and upgrade agents using Puppet Plan on the PE console and CLI
PE version 2023.8.0 introduces Puppet Plan on the PE console and CLI which enables users to install and upgrade agents to intermediate and latest versions without upgrading their PE server.

Platform support

Agent platforms added
This release adds support for the Puppet agent on the following operating system platforms:
  • RedHat Enterprise Linux 9 ppc64le
  • Fedora 40 x86_64
  • Ubuntu 24.04 amd64
  • Ubuntu 24.04 aarch64
  • Amazon Linux 2 aarch64
  • Rocky 9 x86_64
  • Rocky 9 aarch64
  • Alma Linux 9 x86_64
  • Alma Linux 9 aarch64

Resolved issues

Tasks containing a description without any parameters fixed
In PE 2023.7 and PE 2021.7.8, if the task metadata on the Run a task screen in the PE console, contained a description without any parameters, the console did not display the description. This issue has been resolved in PE 2023.8.0 and PE 2021.7.9.
Patching setup in the console no longer allows selection of agentless nodes
In order to receive patches, a node must have an agent installed. However, in PE 2023.7, agentless nodes could be added to patching node groups in the patching setup workflow in the PE console. This issue has been resolved in PE 2023.8.0 and users can no longer selection agentless nodes in the console.
SAML login no longer fails when changing the rbac_token_maximum_lifetime class
When modifying the rbac_token_maximum_lifetime parameter in Node groups > PE Infrastructure in the PE console to anything other than the default of 10y, the user received the following error when trying to use SAML login:
{
  "kind": "puppetlabs.rbac/saml-response-processing-error",
  "msg": "There was an error processing the SAML response: \"No implementation of method: :to-date-time of protocol: #'clj-time.coerce/ICoerce found for class: clojure.lang.Keyword\""
}	

This issue is fixed in PE 2023.8.0 and PE 2021.7.9.

pe-host-action collector service is stopped and restarted during backup restore
In PE 2023.7, the pe-host-action-collector service did not stop and restart during backup restore and subsequently had stale data (usage and license) until the service was restarted. This issue is resolved in PE 2023.8.0.
Create patching group workflow no longer fails to set patch group
In PE versions 2023.3-2023.7, when using the new patching workflow, the workflow correctly created a node group under the Node groups > PE Patch Management. However, the new node group failed to add the class with the patch_group parameter set. This issue has been resolved in PE 2023.8.0 with the class parameters set correctly.
Exec resources failure while using lockless code deploy and applying a compiler’s catalog simultaneously fixed
A race condition that could cause one or more executive resources to fail if a code deploy occurred at the same time as a compiler’s catalog was applied has been fixed.
Reliability of the toggle_lockless_deploys plan fixed
In versions PE 2023.7 and PE 2021.7.8, the toggle_lockless_deploys plan could encounter a race condition when running causing spurious failures. It also would not update Hiera data in the way needed for the lockless deploys setting to be honored on the replica in DR/HA setups. The plan is now more robust and works with DR/HA.
Unable to view a node’s Groups tab in the PE console if view permission is not enabled for any single group the node is in fixed
In versions PE 2023.7 and PE 2021.7.3 - 2021.7.8, if a user did not have permission to view some of the groups their node were in, they could not view their node in any of their node's groups to which they have rights and received an error message stating that they did not have permission to view the group. This issue has been resolved in PE 2023.8 and PE 2021.7.9.
Occasional failure due to a race condition while provisioning a replica fixed
During provisioning of a replica, with either the puppet infra provision replica or puppet infra run enable_ha_failover commands, when the subscription on the replica was established, the Puppet agent did not wait for the subscription initialization to complete and let it run in the background. This resulted in a race condition in which pglogical performed a pg_restore on the database structure while the Puppet agent simultaneously made other database changes. This caused a variety of error signatures, but typically displayed as ERROR: tuple concurrently updated in the PostgreSQL log. Now, the provisioning process waits for the database structure and data to complete its initial sync before proceeding. If you have a large pe-activity database, this may cause provisioning to take a bit longer than usual, up to 10 extra minutes.

PE 2023.7

Released May 2024

Important: Puppet Enterprise (PE) 2023 is our leading-edge PE release stream.

For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z are encouraged to upgrade to either 2021.7 or 2023.

New features

Experience the full value of Puppet Enterprise
If you have installed Puppet Enterprise, you can separately install and use Security Compliance Management (formerly Puppet Comply®) and Continuous Delivery, which are both now covered by your Puppet Enterprise license. Additionally, by purchasing the Puppet Enterprise Advanced license, you can unlock the following premium features:
  • Security Compliance Enforcement (formerly CEM)
  • Advanced Impact Analysis capabilities within Continuous Delivery

For more information about the Puppet Enterprise license, see Getting a license.

Launch Security Compliance Management and Continuous Delivery consoles from the PE console
Starting in PE 2023.7, if you've installed Security Compliance Management and Continuous Delivery, you can launch their respective consoles by clicking quick links in the PE console.

Enhancements

Infrastructure nodes excluded from licensed node count
With the new PE license, your primary server and any deployed database servers, compiler nodes, and replicas are no longer counted towards your licensed node limit. For more information, see How nodes are counted.
Feature toggle for lockless code deploys
If you have enabled Code Manager, you can now turn the lockless code deploys feature on or off by running a puppet infra plan on your primary server. See Toggle lockless code deploys on or off.
Tune file sync performance for lockless code deploys
To help improve the file sync performance for lockless code deploys, two new file sync settings have been added to the puppet_enterprise::master::file_sync class.
  • copy_method: Allows you to specify shell-cp instead of java as the method used by file sync for copying versioned deploys to their directory locations.
  • versioned_sync_pool: Allows you to specify the number of code environments that can be deployed concurrently.
For information about these new settings, see Code Manager settings.
Disaster recovery workflows improved
This release includes improvements to disaster recovery workflows for standard and large installations. The enhancements help to ensure smooth failover to your primary server replica, and minimize potential for disruption in cases where replica promotion is required. See Configuring disaster recovery.
Correct CA directory automatically set up during upgrade
Starting in 2023.7, when you upgrade PE, the installer checks that your certificate authority (CA) directory is set up at /etc/puppetlabs/puppetserver/ca and if necessary, the installer automatically migrates the CA to this directory. This enhancement mitigates the risk of certificate collisions during disaster recovery procedures.
Enhanced logging of schema validation
In the Puppet Server version bundled with PE 2023.7, validation messages in the logs have been improved to provide more context about failed schemas.
Strengthened default password policy
The default password policy for the PE console has been updated to include the following requirements:
  • Passwords must be at least 12 characters in length and must include upper and lowercase letters, special characters, and numbers.
  • The last five previous passwords cannot be reused when passwords are changed.
If you currently have customized Password complexity parameters, your existing configuration will not be overridden when you upgrade.
Strengthened login security
To enhance security for console logins, additional session replay prevention mechanisms were implemented.
Supported ciphers updated
To enhance data security, the list of supported ciphers has been updated. See Compatible ciphers.

Platform support

Agent platforms added
This release adds support for the Puppet agent on the following operating system platforms:
  • Amazon Linux 2023 amd64
  • Amazon Linux 2023 aarch64
  • Debian 11 aarch64
  • Debian 12 amd64
  • Debian 12 aarch64
  • macOS 14 ARM
  • macOS 14 x86_64
  • FIPS 140-2 compliant Red Hat Enterprise Linux (RHEL) 9 x86_64
Client tools platforms added
Support has been added for PE client tools on the following operating system platforms:
  • Amazon Linux 2023 amd64
  • macOS 14 ARM
Solaris 11 packages now verified with GPG
Starting with PE 2023.7 and 2021.7.8, Solaris 11 agent packages are no longer signed with a DigiCert code signing certificate. Instead, you can verify the package's authenticity by using GPG-based verification with the provided .asc file.

Resolved issues

Replica promotion no longer corrupts file sync when lockless code deployment is enabled
In PE versions 2023.0 through 2023.6, and 2021.7.2 through 2021.7.7, if the lockless code deployment feature was enabled, using the disaster recovery workflow to promote a replica could lead to file sync corruption and code deployment failures. The issue is resolved in PE 2023.7 and 2021.7.8.
Fixed issue affecting recover_configuration cron job
In PE versions 2023.6 and 2021.7.7, the recover_configuration cron job could sometimes cause a Puppet Server restart, which in turn could cause an in-process provisioning of a replica to fail. The issue is resolved in PE 2023.7 and 2021.7.8.
Puppet Server includes REXML gem
In PE 2023.5 and 2023.6, Puppet Server did not include the REXML Ruby gem, resulting in problems for modules reliant on XML interactions with the REXML library. The gem is included in the Puppet Server version bundled with PE 2023.7.
Node-pinning issue fixed
In earlier versions of the Puppet Enterprise console, when a node group was set to match any rule, pinning a node resulted in the pinned node rule being incorrectly displayed in the main rules section rather than in the pinned nodes section. This issue is resolved in PE 2023.7.
Backup and restore commands automatically use Puppet binary path
In 2023.6 and 2021.7.7, the puppet backup create and puppet backup restore commands would fail if the PATH variable didn't include the directory with the Puppet binary. This could occur, for example, when running the backup command from a cron job. Now, the full path to the Puppet binary is used automatically by the puppet backup create and puppet backup restore commands.
Security fixes
Addressed the following CVEs:
  • CVE-2024-22871
  • CVE-2024-1597
  • CVE-2024-25710
  • CVE-2024-26308
  • CVE-2023-42503
  • CVE-2024-46218

PE 2023.6

Released February 2024

Important: Puppet Enterprise (PE) 2023 is our leading-edge PE release stream.

For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z are encouraged to upgrade to either 2021.7 or 2023.

New features

Identify operational issues affecting infrastructure nodes
The console now includes an Operational status page showing the result of the latest checks performed by the pe_status_check module. Issues requiring your attention are listed under the affected infrastructure nodes. For more information, see Identify operational issues affecting infrastructure nodes.
Important: If you previously installed the pe_status_check module from the Forge or specified a version in your Puppetfile, ensure that you remove the previously installed version. This allows the latest version bundled with PE to be asserted.

Enhancements

Puppet Server automatically regenerates CRLs that are nearing expiry
Starting in PE 2023.6, Puppet Server performs daily checks on Certificate Revocation Lists (CRLs) and if a CRL is due to expire within 30 days, Puppet Server automatically regenerates it. This enhancement significantly reduces the risk of failed Puppet runs and service outages caused by expired CRLs.
Improved plan concurrency with new pe-plan-runner service
PE 2023.6 introduces the pe-plan-runner service, which improves scalability and performance compared to the existing orchestrator service. Disabled by default, the new service runs on the primary server, allowing concurrent execution of up to 100 plans, with potential for further scaling based on available primary server memory.
With pe-plan-runner enabled, you can continue scheduling and running plans using existing PE console workflows or Orchestrator APIs, and any previously scheduled plans will automatically be executed by pe-plan-runner.
To start running plans over pe-plan-runner instead of the orchestrator service:
  1. Click Node groups > PE Infrastructure > PE Orchestrator.
  2. Select the Classes tab and locate the puppet_enterprise::profile::orchestrator class.
  3. From the Parameter name dropdown, select plan_runner_active and enter true as the value.
  4. Click Add to node group and commit your changes.
Tip: After enabling pe-plan-runner, monitor memory usage on the primary server, as poorly optimized plans may adversely affect performance.
Enhanced workflow for configuring and running task jobs in the console
The process of configuring and running task jobs has been divided into three clear steps in the Tasks section of the console. You can now configure the task job, use one of the three node-targeting methods, and review your setup before scheduling or running the task.
Include or exclude catalog resource edges in catalogs sent to PuppetDB
By default, catalogs submitted to PuppetDB include resource edges, providing data that is useful if you want to identify or analyze the relationships between catalog resources. Starting in PE 2023.6, you can modify the submit_catalog_edges parameter in the puppet_enterprise::profile::master::puppetdb class to exclude resource edges from catalogs. This setting is beneficial if you do not require resource edge data and want to reduce the amount of data stored by PuppetDB.
Specify ciphers to use for connections between PE console and LDAP servers
PE 2023.6 includes a new ldap_cipher_suites parameter in the puppet_enterprise::profile::console class. This parameter allows console users to specify an array of ciphers to use when establishing connections to configured LDAP servers. By default, the value is set to $puppet_enterprise::ssl_cipher_suites, which captures the array of ciphers specified by the puppet_enterprise::ssl_cipher_suites parameter.
Upgraded logback
To address CVE-2023-6378, logback is upgraded to version 1.3.14. If you want to use a customized setting for the logappender variable, see Upgrade cautions for information about avoiding disruptions in logging.
By default, the puppet_enterprise::profile::agent class manages some puppet.conf settings
Starting in PE 2023.6, the manage_puppet_conf parameter in puppet_enterprise::profile::agent class is set to true by default, meaning that all settings configured in the puppet_enterprise::profile::agent class are applied to the puppet.conf file.
If you do not want to use the puppet_enterprise::profile::agent class to manage the puppet.conf file, ensure you set manage_puppet_conf to false.

Platform support

Added agent platforms
Support is added for the following operating system platforms:
  • AIX 7.3

Resolved issues

Upgraded concurrent-ruby to resolve issue that could cause Puppet Server memory leak
A known issue in the concurrent-ruby version packaged with PE 2023.4 and 2023.5 could cause Puppet Server memory leaks, resulting in gradual degradation of Puppet Server performance until the service crashed or was restarted. To resolve this issue, concurrent-ruby is updated to version 1.2.2.
Restoring PE from a backup no longer fails when puppet agent is running
Previously, when running puppet-backup restore, if a Puppet run was either already in progress or started during the restore process, the restore operation could fail with an error. This issue is fixed in PE 2023.6.
Restoring PE from a backup no longer fails if lockless code deployments are enabled
In previous PE versions, running puppet-backup restore resulted in a fatal error if the puppet_enterprise::profile::master::versioned_deploys parameter was set to true. The issue is fixed in PE 2023.6.
Setting the classifier_host parameter no longer causes failure in puppet-backup restore process
In previous versions, the puppet-backup restore process could fail in cases where the puppet_enterprise::profile::master::classifier_host parameter was defined. The issue is fixed in PE 2023.6.
Correct handling of cache collisions in PE console
Previously in the PE console, incorrect handling of cache collisions could result in an HTTP 500 error message showing a lengthy stack trace. The issue is fixed in PE 2023.6.
Deferring the command attribute for exec resources no longer causes catalog errors
Previously, Puppet's autorequire functionality encountered failures when the command attribute of an exec resource was deferred. The issue is fixed in PE 2023.6.
Puppet agent starts as expected on FIPS-compliant RHEL 7 and 8
Previously, on FIPS-compliant Red Hat Enterprise Linux (RHEL) 7 or 8, upgrading the agent to Puppet 8 could cause the puppet service to stop unexpectedly. The issue is fixed in PE 2023.6.
Security fixes
Addressed the following CVEs:
  • CVE-2023-6378
  • CVE-2023-40167
  • CVE-2023-36479
  • CVE-2023-41900
  • CVE-2023-5869
  • CVE-2024-20952
  • CVE-2024-20918
  • CVE-2023-44487
  • CVE-2023-5072
  • CVE-2024-20932
  • CVE-2023-38546

PE 2023.5

Released November 2023

Important: Puppet Enterprise (PE) 2023 is our leading-edge PE release stream (also referred to as STS). For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z are encouraged to upgrade to either 2021.7 or 2023.

Enhancements

Enhanced options for creating fact-based node group rules
When creating fact-based node group rules, you can now include or exclude nodes based on whether a fact, expressed as an array of values, contains a specific value.
For information about creating fact-based rules in the console, see Writing node group rules.
For information on using rules when forming requests to the node classifier API, see Rule condition grammar.
Updated common PQL queries in console
When configuring Puppet runs in the console, you can choose from a range of common Puppet Query Language (PQL) queries to target nodes for jobs and tasks. With the removal of legacy facts in Puppet 8, common queries that used legacy facts have been updated to use equivalent structured facts.

Platform support

Added agent platforms
Support is added for the following operating system platforms:
  • Red Hat Enterprise Linux (RHEL) 9 ARM64
  • Ubuntu 22.04 ARM64

Resolved issues

Fixed issue with puppet_enterprise::profile::master::r10k_known_hosts parameter
In PE 2023.4, if you entered an array of hashes specifying different SSH key "type" values for a single host, failing to include unique "title" values within each hash resulted in a catalog compilation error that prevented r10k and Code Manager from functioning.
In PE 2023.5 the issue is fixed so, in setting the puppet_enterprise::profile::master::r10k_known_hosts parameter, you no longer have to include unique "title" values within each hash when you specify different SSH key types for a single host.
Installing packages with Ubuntu’s Advanced Packaging Tool (APT) no longer causes restarts of pe-puppetserver and pe-orchestration-services
On Ubuntu 22.04, if you use the apt or apt-get commands to install new packages, the needrestart app no longer triggers unexpected restarts of pe-puppetserver and pe-orchestration-services.
Embedded Puppet (EPP) functions now return correctly encoded strings
In PE 2023.4, EPP functions returned binary strings instead of UTF-8 strings. If you used the epp or inline_epp function to generate parameters for exported resources, then the compiler stored the parameter values as base64 encoded strings in PuppetDB. This issue resulted in corrupted data that could not be read or processed when nodes collected the exported resource from PuppetDB. In PE 2023.5, the issue is fixed, and EPP functions now return UTF-8 encoded strings.
Console caching issue resolved
Previously, when adding cache entries, the caching mechanism in the PE console sometimes became stuck in a loop. The issue is resolved in PE 2023.5.
Security fixes
Addressed the following CVEs:
  • CVE-2023-40175
  • CVE-2023-38545
  • CVE-2023-36478
  • CVE-2023-44487
  • CVE-2023-4759
  • CVE-2023-30589
  • CVE-2023-5309

PE 2023.4

Released October 2023

Important: Puppet Enterprise (PE) 2023 is our leading-edge PE release stream (also referred to as STS). For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z are encouraged to upgrade to either 2021.7 or 2023.

CAUTION: PE 2023.4 includes several major component upgrades that introduce breaking changes. Before upgrading, carefully review all Upgrade cautions.

New features

PE certificate authority supports auto-renewal of agent certificates
If your installation includes puppet-agent 8.2.0 or a later version, PE is preconfigured to allow the certificate authority service to generate new agent certificates ahead of certificate expiration dates. This default functionality helps prevent disruption associated with certificate expirations. Optionally, you can turn off auto-renewal of agent certificates and customize your PE certificate authority settings.
Default timeout limits for deploy jobs
Timeout limits forcibly stop deploy jobs that run too long. This feature is useful for stopping jobs that are stuck, without requiring you to manually monitor the progress of jobs.
CAUTION: The feature for forcibly stopping deploy jobs can result in incomplete Puppet runs, partial configuration changes, and other issues. When setting timeout limits, consider the job scope, typical runtime, and your infrastructure's capacity (such as concurrency limits).
The default timeout limit is 30 minutes per node. You can change the global default limit by modifying the default_deploy_node_timeout setting in your Orchestrator and pe-orchestration-services parameters.
View and edit scheduled plans in the console
You can now view and edit scheduled plan details in the console.
View and edit scheduled jobs in the console
You can now view and edit scheduled job details in the console.

Enhancements

Puppet 8 is installed with PE 2023.4
When you install PE 2023.4, an upgraded version of Puppet is installed automatically. Puppet 8 includes several changes that can enhance PE performance capability. For example:
  • Starting in Puppet 8, legacy facts are replaced by structured facts.
  • Strict validation is enabled by default.
  • Ruby is upgraded to version 3.2.
Important: For information about these and other key changes in Puppet 8 that might affect your PE upgrade, see Puppet upgrade in 2023.4 and later.
r10k upgrade
PE includes r10k version 4.0, which has been updated to enhance scalability, reduce dependency risks, and align with Git security best practices.
Important: To review information about changes introduced in r10k 4.0 that might affect your PE upgrade, see Upgrade cautions.
Task concurrency limit now pertains to individual tasks or plans
The task_concurrency setting defines the maximum number of task or plan actions that can be executed simultaneously.
Previously, the concurrency limit pertained globally to actions on any nodes in your installation that were targeted by any current task or plan jobs. Now, the concurrency limit is specific to the nodes targeted by individual task or plan jobs. This improvement significantly reduces latency when multiple task or plan jobs are run simultaneously.
For example, in a company that uses PE, four users each run a task job targeting 10 nodes. The four task jobs are similar in scope and the users initiate their jobs simultaneously. The task_concurrency parameter is set to 10. Previously, with a concurrency limit of 10, task actions would begin executing for one of the jobs and the three remaining jobs would be queued. Now in this scenario, the concurrency limit pertains to each job, so all 40 task actions are executed concurrently. Because the four task jobs are similar in scope, they can be expected to be completed in roughly the same timeframe.
The global default concurrency limit is 1000 actions per job. You can change the global default limit by modifying the task_concurrency parameter value in your Orchestrator and pe-orchestration-services parameters.
Enhanced workflow for configuring and running jobs in the console
The process of configuring and running jobs has been divided into three clear steps in the Jobs section of the console. You can now configure the job, use one of the three node-targeting methods, and review your setup before scheduling or running the job.
Classifier service automatically replaces legacy facts in node group rules
With the removal of legacy facts in Puppet 8, the PE classifier service now analyzes your node group rules and automatically replaces legacy facts with corresponding structured facts. If any of your node group rules contain legacy facts that cannot be directly mapped to structured facts, the classifier service generates warning messages in the logs, prompting you to manually remove or replace the unmappable legacy facts. For more information about the removal of legacy facts in Puppet 8, see Puppet upgrade in 2023.4 and later.
PE installer flags unmappable legacy facts in node group rules
Because legacy facts are removed in Puppet 8, the PE installer now examines your existing node group rules and if any unmappable legacy facts are found, the installation process stops with a warning. To proceed with installation, you can replace or remove unmappable legacy facts and re-run the installer. For more information about the removal of legacy facts in Puppet 8, see Puppet upgrade in 2023.4 and later
Session timeout warning in the PE console
Previously, whenever a console session timed out due to inactivity, users were logged out automatically and returned to the console login screen without warning. Now, whenever a session is about to expire due to inactivity, the console displays a warning modal to inform users they will be logged out soon. The warning modal includes an option to continue the session.
You can configure the behavior of the timeout modal using the following console service parameters:
  • puppet_enterprise::profile::console::session_timeout_polling_frequency_seconds
  • puppet_enterprise::profile::console::session_timeout_warning_seconds
Orchestrator HTTP-client limits can be configured to match infrastructure requirements
You can now specify HTTP-client connection limit parameters in the puppet_enterprise::profile::orchestrator class. You can set connection limits for authenticated and unauthenticated clients by specifying an integer value for the following parameters:
  • max_connections_per_route_authenticated
  • max_connections_total_authenticated
  • max_connections_per_route_unauthenticated
  • max_connections_total_unauthenticated
Orchestrator socket timeout is configurable
By default, whenever no data is available on the socket, the orchestrator waits for a maximum of 120,000 milliseconds before closing the HTTP connection. Now you can specify the maximum time before socket timeout by changing the default value of the socket_timeout parameter in the puppet_enterprise::profile::orchestrator class.
Enhanced logging of certificate authority actions
Previously, agent certificate requests were authorized using the ”pp_cli_auth”: “true” certificate extension. Now, when RBAC tokens are available, token-based authentication is used. This new default authorization method allows better auditability because user IDs that trigger certificate authority actions are reported to the audit log. If you want to configure the certificate authority service settings so that RBAC tokens are always required for authorization of agent certificate requests, you can set the value of allow_puppetlabs_certificate_authentication to false in your certificate_authority service parameters.
More efficient agent run reporting to conserve storage in PuppetDB
Previously, agent run reports submitted to PuppetDB contained significant amounts of data about unchanged managed resources. Now by default, to conserve storage space in PuppetDB, agent run reports only include data relating to changes enforced by the Puppet run. Data about the desired state of each managed resource is still available in agent catalogs. To revert to the previous behavior for agent run reporting, you can modify the puppet_enterprise::profile::agent::exclude_unchanged_resources parameter.
Improvements to error logging for the puppet backup command
Previously, error messages returned by the puppet backup command were generic in many cases. Now, descriptive error messages are displayed both in the terminal and in the log file, and you can use a --debug flag with puppet backup to extend error logging to all underlying Puppet commands.
Optimized translation of classifier rules in PuppetDB queries
Classifier rule translation has been optimized to produce better queries to PuppetDB when regular expressions are used in fact matching.
Restriction: This enhancement does not impact trusted facts, so suboptimal queries can still be produced when regular expressions are used against trusted facts.

Platform support

PE 2023.4 adds support for the following operating system platforms.
Added primary server platforms
Red Hat Enterprise Linux (RHEL) 9 x86_64
Ubuntu 22.04 amd64
Added agent platforms
macOS 13 ARM and x86_64
Added client tools platforms
macOS 13 ARM and x86_64
With this release, support was removed for several previously deprecated platforms. Before upgrading, review the following list of removed platforms and the important information in Platforms removed in 2023.0 and later.
Removed agent platforms
AIX 7.1
CentOS 6
CentOS 7 aarch64
macOS 10.15
Oracle Linux 6
Oracle Linux 7 aarch64
Red Hat Enterprise Linux (RHEL) 6
Red Hat Enterprise Linux (RHEL) 7 aarch64
Scientific Linux 6
Scientific Linux 7 aarch64
Solaris 10
Removed client tool platforms
CentOS 6
macOS 10.15
Oracle Linux 6
Red Hat Enterprise Linux (RHEL) 6
Scientific Linux 6

Deprecations and removals

Removed platforms
For information about platforms removed in this release, see the Platform support section.
Puppet 8 deprecations and removals
For information about deprecations and removals associated with the upgrade to Puppet 8, see Puppet upgrade in 2023.4.

Resolved issues

Installing Windows agent through the console no longer fails when option to test connections is selected
In PE 2021.2 and later, when installing Windows agents in the console’s Install agent on nodes screen, checking the Test Connections checkbox before clicking Add nodes caused the process to hang indefinitely. The issue is resolved in PE.
Security fixes
Addressed CVE-2023-5255

PE 2023.2

Released June 2023

Important: Puppet Enterprise (PE) 2023 is our leading-edge PE release stream (also referred to as STS). For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z are encouraged to upgrade to either 2021.7 or 2023.

Resolved issues

Security fix
Addressed CVE-2023-2530

PE 2023.1

Released May 2023

Important: Puppet Enterprise (PE) 2023 is our leading-edge PE release stream (also referred to as STS). For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z are encouraged to upgrade to either 2021.7 or 2023.

Enhancements

Improved performance when querying PuppetDB
This enhancement helps to improve performance for PuppetDB queries that contain large arrays, for example, if many nodes are enumerated or many terms are joined by a single "and" or "or" element.
Improved performance for the each, map, and filter functions in the Puppet language
Previously, the Puppet language built-in functions each, map, and filter showed poor performance and consumed unnecessary resources when run on JRuby software. The issue was resolved to enhance performance.
Puppet Server provides more reliable warnings when it cannot check for an update
By default, Puppet Server periodically checks whether a new version of Puppet Server is available. Previously, if Puppet Server could not connect to the update server, users were not provided with adequate information about the error. Starting with Puppet Server 7.10.1, a warning about the error is available in the log file.

Deprecations and removals

Deprecated PSON
In previous releases, Pure JavaScript Open Notation (PSON) was used in Puppet to serialize data for transmission.

PSON is deprecated in Puppet 7 and will be removed in Puppet 8.

Resolved issues

Tasks page is available following a software update
After upgrading PE from 2019.8 to 2021.7.1, the Tasks overview page in the PE console sometimes failed to load because of a timeout error. The issue is fixed in PE 2021.7.3 and 2023.1.
Scheduled task jobs run successfully without a defined timeout
In PE 2023.0, task jobs failed to start if they were scheduled without an explicitly defined timeout. In PE 2023.1, the issue is resolved to help ensure that task jobs start as scheduled even without an explicitly specified timeout option. If a timeout option is not explicitly defined, the default timeout for tasks is applied.
Timeout and concurrency values for scheduled tasks can be viewed and edited in the console
In PE 2023.0, the timeout and concurrency values for a scheduled task could not be viewed or edited in the PE console. This issue is fixed in PE 2023.1:
  • When you view a scheduled task in the console, any specified timeout and concurrency values are displayed in the new Timeout and Concurrency fields.
  • When you edit a scheduled task in the console, you can update the values in the new Timeout and Concurrency fields.
  • Any timeout or concurrency values that you specify for scheduled tasks will be applied.
When tasks are rerun in the console, timeout and concurrency attributes are preserved
In PE 2023.0, tasks that were rerun in the PE console did not properly preserve the concurrency and timeout attributes of the task job. This issue is fixed in PE 2023.1.
Access rights for remote users can be revoked and reinstated from the console
In PE 2023.0, a defect was introduced that prevented the revocation or restoration of some remote users by using the PE console. This issue is resolved in PE 2023.1.
Performance issue with Puppet agent runtimes is resolved
After an upgrade from PE 2019.8.12 to PE 2021.7.1, some users saw a significant increase in Puppet agent runtimes. The increase was caused by Facter 4, which was not using cached information to resolve facts. As a result, facts were resolved multiple times. The issue is now resolved to normalize the performance of the Puppet agent.
Enabling the lockless code deploy feature no longer causes performance issues in PuppetDB catalog compilation
When the versioned_deploys setting is enabled, Puppet previously reported the full directory path to the environment after resolving symbolic links as the source for resources in a catalog. Puppet now reports the path to the resource before resolving symbolic links in the environment path to help prevent instability of the PuppetDB instance.
Certificates and keys can be backed up and restored by specifying the certs scope
Previously, if you ran the puppet-backup create command and specified a scope of certs, the command failed to back up the certificate authority root key and certificates. This issue occurred because Puppet 7 introduced a new default path for the certificate authority (CA) directory (/etc/puppetlabs/puppetserver/ca), but the puppet-backup create command failed to locate the new directory. Similarly, if you ran the puppet-backup restore command with a scope of certs, the restore operation failed. The CA directory issue is resolved so that backup and restore operations can run successfully.
Timeouts can be specified for SAML authentication
Previously, when users configured the PE console to specify session-timeout and session-maximum-lifetime values, the settings were applied to Lightweight Directory Access Protocol (LDAP) tokens and local login tokens. However, the specified settings were not applied to Security Assertion Markup Language (SAML) tokens, which are used for authentication with SAML identity providers. The issue is corrected to ensure that the specified settings also apply to SAML session lifetimes.
Updates implemented to help users enter valid URLs
In previous versions of PE, the role-based access control (RBAC) service permitted the entry of invalid URLs when users specified the Organizational URL setting. Login attempts would then fail with the following error message:
'Invalid settings: organization_not_enough_data'

In PE 2021.7.3 and 2023.1, the RBAC service is updated to enforce valid URLs when users create or update a connection to a SAML identity provider, and the PE console displays a warning if the user enters an invalid URL for the Organizational URL setting.

User-defined temporary directory is honored during PE restore operations
After you back up your PE infrastructure, you can use the puppet-backup restore command to restore the backup. Previously, if you set the —tmpdir flag or the TMPDIR environment variable to specify a temporary directory for restore operations, the directory was not honored, and the default /tmp directory was used in some cases. In addition, some files were not cleaned up after the restore operation. This issue is corrected to ensure that the user-specified directory is used, and all temporary files are removed after the restore operation.
Issue that caused an unexpected increase in CPU usage is resolved
In PE 2021.7.1, 2021.7.2, and 2023.0, an issue with Puppet Server caused an unexpected increase in central processing unit (CPU) usage in some environments. CPU usage continued to grow and some operations took longer than expected until the Puppet Server service was restarted. This issue is resolved in PE 2023.1 and 2021.7.3.
Security fixes
Addressed CVE-2023-1894 and CVE-2023-26048.

PE 2023.0

Released January 2023

Important: PE 2023 is our new leading-edge PE release stream (also referred to as STS). For important information about upgrading to 2023, see Upgrading Puppet Enterprise.

If you're on the LTS stream (2021.7), you'll find release notes and other information for that series in the 2021.7 documentation.

Customers on 2019.8.z, which is EOL, are encouraged to upgrade to either 2021.7 or 2023.

New features

Authenticate users in multiple LDAP domains
You can now connect multiple Lightweight Directory Access Protocol (LDAP) domains to PE. This new feature brings many changes to the role-based access control (RBAC) API and LDAP-related pages in the PE console.
In the PE console, view and manage all of your LDAP external directory service connections on the LDAP tab of the Access control page.
The Test connection button is removed. When you Connect to external directory services, the Connect button now automatically tests the connection before saving the configuration.
Use the Certificate chain field (or cert_chain API key) to define unique certificate chains across servers.
The following new endpoints replace deprecated or removed endpoints. For a list of deprecated and removed endpoints, refer to the Deprecations and removals section of these release notes.
Responses from these endpoints now include the identity_provider_id:
Default timeout limits for tasks and plans
Timeout limits forcibly stop tasks and plans that run too long. This feature is useful for stopping tasks and plans that are stuck without requiring you to manually monitor task or plan progress.
CAUTION: The feature for forcibly stopping tasks and plans can result in incomplete Puppet runs, partial configuration changes, and other issues. When setting timeout limits, consider the task or plan scope, typical runtime, and your infrastructure's capacity (such as concurrency limits).
The default timeout limits are 40 minutes for tasks (per node) and 60 minutes for plans (for the entire plan run). You can change the global default limits by modifying the default_task_node_timeout and default_plan_timeout settings in your Orchestrator and pe-orchestration-services parameters.
Alternatively, you can set timeout limits for an individual task or plan when Running tasks from the console, Running plans from the console, or running tasks and plans with the Orchestrator API.
You can use the timeout option with the following Orchestrator API endpoints:
Unique status for queued jobs
To better differentiate queued-but-unstarted jobs from jobs that are running, a new pending state was introduced for queued jobs.
The pending state is visible in the console and in responses from GET /plan_jobs and GET /plan_jobs/<job-id>.
View and edit scheduled tasks in the console
You can now view and edit scheduled task details in the console.

Enhancements

Java 17 upgrade
This version upgrades Java from version 11 to 17 and changes the default garbage collector from Parallel to G1.
Thoroughly test PE 2023.0 in a non-production environment before upgrading if you customized PE Java services or you use plug-ins that include Java code.
Stop in-progress plans in the console
When Running plans in PE, you can click Stop plan on the plan's run details page to stop the plan. In this way, you can prevent new tasks from starting and allow in-progress tasks to finish. To forcibly stop in-progress tasks from a stopped plan, follow the instructions in Stop a task in progress.
Forcibly stop in-progress tasks in the console
To Stop a task in progress, you can now both stop and forcibly stop in-progress tasks from the console. Previously, you had to use the Orchestrator API to forcibly stop tasks.
CAUTION: A forcible stop is the last resort when a task is stuck. This type of stop can result in incomplete Puppet runs, partial configuration changes, and other issues.
Provisioning replicas requires matching agent versions
When provisioning a replica, the target node's agent version must match the primary server's agent version. If the versions don't match, the puppet infra provision replica command fails before initializing the provisioning process. Previously, the agent version wasn't checked, and mismatched agent versions caused provisioning to fail partway through.
Increased task_concurrency limit
The default value of the task_concurrency orchestrator parameter was increased from 250 to 1000.
recover_configuration command recreates nodes files
Previously, the puppet infrastructure recover_configuration command merged new values into the nodes files (at /etc/puppetlabs/enterprise/conf.d/nodes) instead of overwriting the files. This process caused problems if you deleted a value relevant to one or more nodes, because the deleted value would remain in these files and continue to be applied.
Now, the recover_configuration command fully rewrites the nodes files on each invocation. This process matches how the command handles changes to the user_data.conf file.
Notification when session expires due to inactivity
PE redirects users to the login page when a session expires due to inactivity. When this happens, the login page now includes a message that indicates why the user was logged out.
Improved performance when regenerating agent certificates for multiple agents
The puppet infrastructure run regenerate_agent_certificate action is now faster when you Regenerate agent certificates for multiple agents. You can also now use the agent_pdb_query parameter to use a PDB query to generate a list of agents for which you want to regenerate certificates.
This action now uses the Puppet Server CA API endpoints directly, rather than relying on the puppetserver ca CLI, as it did previously. This process is faster, but, if you encounter problems, you can revert to the previous behavior by including use_puppetserver_cli=true in the command.
Specify Code Manager worker cache cleanup interval
The deploy_pool_cleanup_interval specified how often workers pause to clean their on-disk caches. Learn more about this setting in Code Manager parameters.
This release includes enhancements to cipher compatibility. For a complete list, go to Compatible ciphers.
CHACHA20 ciphers, compatible with non-FIPS PE installs
TLS_CHACHA20_POLY1305_SHA256 (TLSv1.3)
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (TLSv1.2)
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (TLSv1.2)
TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (TLSv1.2)
AES versions of two GCM ciphers, compatible with FIPS and non-FIPS installs
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (TLSv1.2)
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (TLSv1.2)
Removed restrictions
TLS_CHACHA20_POLY1305_SHA256 is no longer limited to Bolt server, ACE server, and NGINX.
ECDHE-ECDSA-CHACHA20-POLY1305 is no longer limited to NGINX.
ECDHE-RSA-CHACHA20-POLY1305 is no longer limited to NGINX.

Platform support

With this release, several previously deprecated platforms were removed. Before upgrading, review the important information provided in Platforms removed in 2023.0 and later.
Removed primary server platforms
CentOS 8
Removed agent platforms
CentOS 8
Debian 9
Fedora 32
Fedora 34
Ubuntu 16.04
Removed patch management platforms
Debian 9
Fedora 34

Deprecations and removals

Deprecated RBAC API endpoints
POST /v1/groups and POST /v2/groups are replaced by POST /command/groups/create.
PUT /v1/ds is replaced by POST /command/ldap/create, POST /command/ldap/update, and POST /command/ldap/delete.
GET /v2/ds is replaced by GET /ldap.
GET /ds/test and PUT /ds/test are replaced by POST /command/ldap/test.
Removed RBAC API endpoints
Removed the previously deprecated GET /v1/ds/, which is replaced by GET /ldap.
Removed platforms
For information about platforms removed in this release, see the Platform Support section.

Resolved issues

Code Manager respects full_deploy setting in Hiera
The full_deploy parameter is now correctly applied when you Customize Code Manager configuration in Hiera.
Previously, full_deploy was disregarded when included in your Code Manager configuration in Hiera. As a work-around, you could create a separate .conf file to manually manage this parameter.
Important: If you created a .conf file for the full_deploy parameter, you must remove this file and reconfigure the parameter in Hiera (as described in Configuring module deployment scope).
Certain plans correctly restore puppet service to pre-plan state
Due to a bug introduced in PE 2021.6, some plans that must stop the puppet service while the plans run were not restoring the puppet service to its pre-plan state after the plan finished running.
The four affected plans, and their associated puppet infra commands, are as follows:
  • The secondary_cert_regen plan, which is triggered by puppet infra run regenerate_compiler_certificate and puppet infra run regenerate_replica_certificate
  • The convert_legacy_compiler plan, which is triggered by puppet infra run convert_legacy_compiler
  • The reprovision_replica plan, which is triggered specifically by puppet infra upgrade replica --only-recreate-databases
  • The enable_ha_failover plan, which is triggered by puppet infra run enable_ha_failover
Important: If you were running PE 2021.6, 2021.7.0, or 2021.7.1 before upgrading to 2023.0, and you ran any of these four plans while running 2021.6, 2021.7.0, or 2021.7.1, check the state of the puppet service on your infrastructure nodes.
PuppetDB database user can purge reports
An issue was fixed to ensure that the PuppetDB database user can purge reports.
Corrected fact list handling in some PE console UI components
Some UI components in the PE console use fact lists. A recent change caused these component to use the entire list of fact names, which caused performance problems in environments with many facts. The handling of fact lists was corrected to fix this issue and improve performance.
Orchestrator code directories excluded from puppet-backup create --scope=config
When Customize scope of backup and restore, the orchestrator code directories (specifically /opt/puppetlabs/server/data/orchestration-services/data-dir and /opt/puppetlabs/server/data/orchestration-services/code) are excluded when you specify the config scope.
These directories are included in the code scope.
Plan action jobs have user data
Previously, jobs started as a result of plan action function didn't have an associated user stored in the database, which caused problems with some orchestrator commands. Now, user data is stored for these jobs.
Garbage collection log fixes
The introduction of Java 11 resulted in two issues relating to garbage collection logs. The issues are now fixed:
Dates and times are now included in garbage collection logs.
The maximum volume of retained garbage collection logs is 256 MB.
Security fixes
Addressed CVE-2022-41946 and CVE-2022-41404.