PE known issues

Puppet Server JVM segfaults during RHEL 7 to RHEL 8 migration

During migration of PE from Red Hat Enterprise Linux (RHEL) 7 to RHEL 8, if your puppetlabs-stdlib module version is older than 9.1.0, the puppetserver Java Virtual Machine (JVM) might experience segfaults at seemingly random intervals. This issue is caused by underlying logic related to password hashing. To avoid this problem, upgrade the puppetlabs-stdlib module to version 9.1.0 or newer.

Converting legacy compilers fails with an external certificate authority

Agent_cert_regen: ERROR: Failed to regenerate agent certificate on node <compiler-node.domain.com>
Agent_cert_regen: bolt/run-failure:Plan aborted: run_task 'enterprise_tasks::sign' failed on 1 target
Agent_cert_regen: puppetlabs.sign/sign-cert-failed Could not sign request for host with certname <compiler-node.domain.com> using caserver <master-host.domain.com>

Converted compilers can slow PuppetDB in multi-region installations

In configurations that rely on high-latency connections between your primary servers and compilers – for example, in multi-region installations – converted compilers running the PuppetDB service might experience significant slowdowns. If your primary server and compilers are distributed among multiple data centers connected by high-latency links or congested network segments, reach out to Support for guidance before converting legacy compilers.

Provisioning a replica may occasionally fail due to a race condition

During provisioning of a replica, with either the puppet infra provision replica or puppet infra run enable_ha_failover commands, when the subscription on the replica is established, the Puppet agent does not wait for the subscription initialization to complete and lets it run in the background. This can result in a race condition in which pglogical is performing a pg_restore on the database structure while the Puppet agent is simultaneously making other database changes. This can cause a variety of error signatures, but typically shows as ERROR: tuple concurrently updated in the PostgreSQL log. However, running into this issue is relatively rare. If you run into this issue:

1. Run puppet infra run reinitialize replica on the replica node.
2. Unpin the replica from the PE Infrastructure Agent node group.
3. Run puppet agent -t on the primary and replica until no changes are observed.
4. Wait for puppet infra status to show all services are operating normally.
5. Run puppet infra enable replica if desired.

FIPS-enabled PE 2023.0 and later can't use the default system cert store

FIPS-compliant builds running PE 2023.0 and later can't use the default system cert store, which is used automatically with some reporting services. This setting is configured by the report_include_system_store Puppet parameter that ships with PE.

Removing the puppet-cacerts file (located at /opt/puppetlabs/puppet/ssl/puppet-cacerts) can allow a report processor that eagerly loads the system store to continue with a warning that the file is missing.

If HTTP clients require external certs, we recommend using a custom cert store containing only the necessary certs. You can create this cert store by concatenating existing pem files and configuring the ssl_trust_store Puppet parameter to point to the new cert store.

Puppet Server FIPS installations don’t support Ruby’s OpenSSL module

FIPS-enabled PE installations don't support extensions or modules that use the standard Ruby Open SSL library, such as hiera-eyaml. As a workaround, you can use a non-FIPS-enabled primary server with FIPS-enabled agents, which limits the issue to situations where only the primary uses the Ruby library. This limitation does not apply to versions 1.1.0 and later of the splunk_hec module, which supports FIPS-enabled servers. The FIPS Mode section of the module's Forge page explains the limitations of running this module in a FIPS environment.

puppet infrastructure tune fails with multi-environment environmentpath

The puppet infrastructure tune command fails if environmentpath (in your puppet.conf file) is set to multiple environments. To avoid the failure, comment out this setting before running this command. For details about the environmentpath setting, refer to environmentpath in the open source Puppet documentation.

Restarting or running Puppet on infrastructure nodes can trigger an illegal reflective access operation warning

When restarting PE services or performing agent runs on infrastructure nodes, you might see this warning in the command-line output or logs: Illegal reflective access operation ... All illegal access operations will be denied in a future release

These warnings are internal to PE service components and have no impact on their functionality. You can safely disregard them.

pe-host-action-collector service is not stopped and restarted during backup restore

The pe-host-action-collector service is not stopped and restarted during backup restore and subsequently will have stale data (usage and license) until the service is restarted. As a workaround, you can restart the pe-host-action-collector service: systemctl restart pe-host-action-collector after doing the restore.

Node groups visibility issue for non-admin users

In PE version 2023.1 and later, non-admin console users might encounter an error when trying to view the groups of a node. This happens if a node is part of multiple groups and the user doesn't have permission to view some of those groups. In such cases, when clicking on the node's Groups tab, users see an error message indicating they lack the required permissions.

Patching setup in the console allows selection of agentless nodes

Currently in the patching setup workflow in the console, agentless nodes can be added to patching node groups. However, in order to receive patches, a node must have the agent installed.

The Configure nodes step of the workflow skips any agentless nodes added in the Create node group step, though the agentless nodes remain pinned to the created patching node group under Node groups > PE Patch Management.

If a patching node group that includes only agentless nodes is created, running Puppet on the Configure nodes step of the workflow fails entirely, though the created patching node group remains under Node groups > PE Patch Management.

Until a fix is implemented, avoid adding agentless nodes in the Create node group step of the patching setup workflow.

Patching fails with excluded YUM packages

In the patching task or plan, using yum_params to pass the --exclude flag in order to exclude certain packages can result in task or plan failure if the only packages requiring updates are excluded. As a workaround, use the versionlock command (which requires installing the yum-plugin-versionlock package) to lock the packages you want to exclude at their current version. Alternatively, you can fix a package at a particular version by specifying the version with a package resource for a manifest that applies to the nodes to be patched.

Create Patching group workflow fails to set patch group

When using the new patching workflow, the workflow correctly creates a node group under Node groups > PE Patch Management. However, the new node group fails to add the class with the patch_group parameter set. This is an important behavior because the UI allows the user to filter based on the patch_group value. Each node group should add the class with the patch_group set to the name of the node group. This will allow filtering on the Available patches screen based on patch_group.

Changing a file type in a control repo produces a checkout conflict error

Changing a file type in a control repository – for example, deleting a file and replacing it with a directory of the same name – generates the error JGitInternalException: Checkout conflict with files accompanied by a stack trace in the Puppet Server log. As a workaround, deploy the control repo with the original file deleted, and then deploy again with the replacement file or directory.